2024-03-29T11:06:53.547 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:06:53.547 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/orig.config.yaml 2024-03-29T11:06:53.581 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:06:53.668 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622685' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi019.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGmST/JS+fhMJ4n0lfob5aItaHmAVSIgXeMLzvsxBGyZM1xYrIRbGnPndg+PnnS6xedpMCjj7De3Od6LJY6fDJo= smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAoMD5AnRoAI8E0Aydsy5qWPX4rpSqlIGnpKho++nzwrit0iEO/vBeomdfg2KdU0flCdTWjSvFvqagARkIeHhmc= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:06:53.668 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:06:53.669 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:06:53.669 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:06:53.670 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:06:53.694 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:06:53.694 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:06:53.694 INFO:teuthology.packaging:ref: None 2024-03-29T11:06:53.694 INFO:teuthology.packaging:tag: None 2024-03-29T11:06:53.694 INFO:teuthology.packaging:branch: main 2024-03-29T11:06:53.695 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:53.695 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:06:54.083 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:06:54.084 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:06:54.095 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:06:54.095 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:06:54.118 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:06:54.131 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:06:54.142 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:06:54.168 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:05.708424', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGmST/JS+fhMJ4n0lfob5aItaHmAVSIgXeMLzvsxBGyZM1xYrIRbGnPndg+PnnS6xedpMCjj7De3Od6LJY6fDJo='} 2024-03-29T11:06:54.189 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:05.710108', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAoMD5AnRoAI8E0Aydsy5qWPX4rpSqlIGnpKho++nzwrit0iEO/vBeomdfg2KdU0flCdTWjSvFvqagARkIeHhmc='} 2024-03-29T11:06:54.189 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:06:54.197 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:06:54.197 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:06:54.197 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:06:54.275 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f50bd05b0d0>, signals=[15]) 2024-03-29T11:06:54.275 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:06:54.284 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:06:54.284 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:06:54.285 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:54.391 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:06:54.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:54.497 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:06:54.505 DEBUG:teuthology.orchestra.run.smithi019:> uname -m 2024-03-29T11:06:54.522 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64 2024-03-29T11:06:54.523 DEBUG:teuthology.orchestra.run.smithi019:> cat /etc/os-release 2024-03-29T11:06:54.580 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="CentOS Stream" 2024-03-29T11:06:54.580 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="9" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:ID="centos" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="9" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:54.581 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:54.582 INFO:teuthology.orchestra.run.smithi019.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:54.582 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server 2024-03-29T11:06:54.611 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-03-29T11:06:54.629 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-03-29T11:06:54.629 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-03-29T11:06:54.686 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-03-29T11:06:54.686 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:06:54.687 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:06:54.688 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-03-29T11:06:54.711 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:06:54.723 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:06:54.756 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:06:54.756 DEBUG:teuthology.orchestra.run.smithi019:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:54.758 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:06:54.775 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:06:54.841 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:06:54.841 DEBUG:teuthology.orchestra.run.smithi019:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:54.843 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:06:54.871 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:06:54.958 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:06:54.975 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:06:54.975 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:06:54.975 DEBUG:teuthology.orchestra.run.smithi019:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:54.976 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:06:54.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:54.993 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-29T11:06:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:06:54.995 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-03-29T11:06:55.051 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:55.051 INFO:teuthology.task.kernel:Running kernel on smithi019: 5.14.0-361.el9.x86_64 2024-03-29T11:06:55.051 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum install -y kernel 2024-03-29T11:06:55.053 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:06:55.053 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.14.0-361.el9.x86_64 2024-03-29T11:06:55.053 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2024-03-29T11:06:57.268 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:06:57.273 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:06:58.693 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2024-03-29T11:06:58.723 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-29T11:07:07.916 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 909 kB/s | 8.0 MB 00:09 2024-03-29T11:07:08.197 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 903 kB/s | 8.0 MB 00:09 2024-03-29T11:07:09.909 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-29T11:07:10.664 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T11:07:13.976 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-29T11:07:14.806 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-29T11:07:15.815 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-29T11:07:16.424 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-29T11:07:17.145 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:07:18.850 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-29T11:07:22.261 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-03-29T11:07:24.000 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-29T11:07:24.602 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:07:24.602 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:07:24.691 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:07:24.696 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:07:24.696 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:07:24.696 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:07:24.696 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout:Install 4 Packages 2024-03-29T11:07:24.697 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:07:24.698 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 76 M 2024-03-29T11:07:24.698 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 123 M 2024-03-29T11:07:24.698 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:07:24.905 INFO:teuthology.orchestra.run.smithi062.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 929 kB/s | 192 kB 00:00 2024-03-29T11:07:25.746 INFO:teuthology.orchestra.run.smithi062.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-03-29T11:07:26.399 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:07:26.399 INFO:teuthology.orchestra.run.smithi019.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:07:26.486 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:07:26.493 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:07:26.493 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout:Install 4 Packages 2024-03-29T11:07:26.494 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:07:26.495 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 76 M 2024-03-29T11:07:26.495 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 123 M 2024-03-29T11:07:26.495 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:07:26.572 INFO:teuthology.orchestra.run.smithi062.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 18 MB/s | 34 MB 00:01 2024-03-29T11:07:26.668 INFO:teuthology.orchestra.run.smithi019.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:07:26.855 INFO:teuthology.orchestra.run.smithi062.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 14 MB/s | 28 MB 00:01 2024-03-29T11:07:26.858 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:26.859 INFO:teuthology.orchestra.run.smithi062.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-29T11:07:26.859 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:07:27.185 INFO:teuthology.orchestra.run.smithi019.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 22 MB/s | 15 MB 00:00 2024-03-29T11:07:27.300 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:07:27.301 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:07:27.735 INFO:teuthology.orchestra.run.smithi019.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T11:07:28.135 INFO:teuthology.orchestra.run.smithi019.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 19 MB/s | 28 MB 00:01 2024-03-29T11:07:28.138 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:28.139 INFO:teuthology.orchestra.run.smithi019.stdout:Total 47 MB/s | 76 MB 00:01 2024-03-29T11:07:28.139 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:07:28.173 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:07:28.173 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:07:28.568 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:07:28.569 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:07:29.449 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:07:29.449 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:07:30.726 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:07:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:31.422 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:31.881 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:07:32.510 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:07:32.596 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:34.326 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:34.457 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:35.486 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:07:35.621 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:39.539 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:39.581 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:40.695 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:07:40.737 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:44.678 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:07:45.837 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:15.568 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:15.569 INFO:teuthology.orchestra.run.smithi062.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:15.569 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:08:15.613 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:16.007 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:16.007 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:16.007 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:16.007 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:16.410 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:16.410 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:08:16.410 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:08:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:08:16.411 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:08:16.609 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:16.609 INFO:teuthology.orchestra.run.smithi019.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:08:16.609 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:08:16.653 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:16.729 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:16.832 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:16.832 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:16.833 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:16.833 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:16.833 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:16.833 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:08:16.833 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:16.833 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:16.916 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:16.917 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:16.917 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 68.el9 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-03-29T11:08:16.955 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8264254 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:16.956 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:16.957 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:16.957 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-03-29T11:08:16.957 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:08:16.957 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:16.958 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:16.958 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:16.958 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:17.163 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:17.164 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:17.164 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:17.164 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:17.547 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:08:17.599 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:08:17.600 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:08:17.958 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:18.057 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:18.057 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:18.057 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:08:18.057 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:08:18.057 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:08:18.057 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:08:18.058 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi019.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:08:18.058 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:08:18.141 INFO:teuthology.orchestra.run.smithi019.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:08:18.141 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:08:18.141 DEBUG:teuthology.orchestra.run.smithi019:> sudo rpm -qi grub2-tools 2024-03-29T11:08:18.189 INFO:teuthology.orchestra.run.smithi019.stdout:Name : grub2-tools 2024-03-29T11:08:18.189 INFO:teuthology.orchestra.run.smithi019.stdout:Epoch : 1 2024-03-29T11:08:18.189 INFO:teuthology.orchestra.run.smithi019.stdout:Version : 2.06 2024-03-29T11:08:18.189 INFO:teuthology.orchestra.run.smithi019.stdout:Release : 68.el9 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Architecture: x86_64 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Group : Unspecified 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Size : 8264254 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:License : GPLv3+ 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Packager : builder@centos.org 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Vendor : CentOS 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:08:18.190 INFO:teuthology.orchestra.run.smithi019.stdout:Summary : Support tools for GRUB. 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout:Description : 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout:hardware devices. 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:08:18.191 INFO:teuthology.orchestra.run.smithi019.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:08:18.192 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:08:18.193 INFO:teuthology.task.kernel:Updating grub on smithi019 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:08:18.193 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:08:18.784 INFO:teuthology.orchestra.run.smithi019.stderr:Generating grub configuration file ... 2024-03-29T11:08:18.980 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:19.010 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-03-29T11:08:19.012 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-03-29T11:08:19.027 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.inssSplttB 2024-03-29T11:08:19.027 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.inssSplttB 2024-03-29T11:08:19.100 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.inssSplttB 2024-03-29T11:08:19.246 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.inssSplttB is 6KB 2024-03-29T11:08:19.295 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.inssSplttB 2024-03-29T11:08:19.309 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:19.379 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:19.513 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:19.513 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:19.513 INFO:teuthology.orchestra.run.smithi062.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:19.515 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:19.672 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-03-29T11:08:19.768 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-29T11:08:20.238 INFO:teuthology.orchestra.run.smithi019.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:08:20.254 INFO:teuthology.orchestra.run.smithi019.stderr:done 2024-03-29T11:08:20.256 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-29T11:08:20.271 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.4NoheBHem0 2024-03-29T11:08:20.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4NoheBHem0 2024-03-29T11:08:20.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.4NoheBHem0 2024-03-29T11:08:20.491 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.4NoheBHem0 is 6KB 2024-03-29T11:08:20.539 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.4NoheBHem0 2024-03-29T11:08:20.553 DEBUG:teuthology.orchestra.run.smithi019:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:08:20.623 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:08:20.623 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:08:20.624 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:08:20.624 INFO:teuthology.orchestra.run.smithi019.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:08:20.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:08:20.785 DEBUG:teuthology.orchestra.run.smithi019:> sudo shutdown -r now 2024-03-29T11:08:49.707 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:08:49.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-29T11:08:49.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:50.789 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:08:50.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-29T11:08:50.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:08.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-03-29T11:09:09.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.19 2024-03-29T11:09:17.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-29T11:09:17.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:18.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-29T11:09:18.281 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:09:20.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-03-29T11:09:32.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-29T11:09:32.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:04.227 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-03-29T11:10:04.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-29T11:10:04.637 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:04.637 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-03-29T11:10:04.652 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:04.653 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:04.653 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:04.653 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:05.654 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:10:05.654 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:05.717 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-03-29T11:10:05.759 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:18.282 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:10:30.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-29T11:10:30.284 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:30.568 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-29T11:10:30.941 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-29T11:10:30.941 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:10:30.941 DEBUG:teuthology.orchestra.run.smithi019:> uname -r 2024-03-29T11:10:30.956 INFO:teuthology.orchestra.run.smithi019.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:10:30.956 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:10:30.956 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:10:30.956 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:10:31.957 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:10:31.957 DEBUG:teuthology.orchestra.run.smithi019:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:10:32.017 INFO:teuthology.orchestra.run.smithi019.stdout:ttyS1 2024-03-29T11:10:32.053 DEBUG:teuthology.parallel:result is None 2024-03-29T11:10:32.054 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:10:32.064 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:10:32.064 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:32.067 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:10:32.087 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:10:32.096 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:10:32.104 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:10:32.105 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:32.123 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:10:32.159 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:10:32.166 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:10:32.166 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:32.186 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:10:32.221 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:32.235 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:32.235 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:32.250 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:10:32.252 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:10:32.258 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:10:32.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:32.279 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:10:32.323 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:10:32.332 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:10:32.332 DEBUG:teuthology.orchestra.run.smithi019:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:32.348 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:10:32.380 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:32.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:32.493 DEBUG:teuthology.orchestra.run.smithi019:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:32.575 DEBUG:teuthology.orchestra.run.smithi019:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:32.617 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:10:32.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:32.685 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:32.724 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:10:32.786 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:32.882 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:10:32.925 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:10:32.925 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:10:32.998 DEBUG:teuthology.orchestra.run.smithi019:> sudo service rsyslog restart 2024-03-29T11:10:33.004 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-03-29T11:10:33.046 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:33.097 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:10:33.467 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:10:33.479 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:10:33.479 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:10:33.499 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:10:33.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo service auditd rotate 2024-03-29T11:10:33.573 INFO:teuthology.orchestra.run.smithi019.stdout:Rotating logs: 2024-03-29T11:10:33.575 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-03-29T11:10:33.637 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-03-29T11:10:33.639 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:10:33.639 DEBUG:teuthology.orchestra.run.smithi019:> /usr/sbin/getenforce 2024-03-29T11:10:33.671 INFO:teuthology.orchestra.run.smithi019.stdout:Permissive 2024-03-29T11:10:33.671 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-03-29T11:10:33.691 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-03-29T11:10:33.691 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi019.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi062.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:10:33.691 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:33.740 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:10:33.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:10:33.768 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:10:33.769 DEBUG:teuthology.orchestra.run.smithi019:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:10:33.823 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:10:33.859 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:10:33.879 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:10:33.896 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:10:33.897 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi019.front.sepia.ceph.com,smithi062.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:15:23.397 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi019.front.sepia.ceph.com'), Remote(name='ubuntu@smithi062.front.sepia.ceph.com')] 2024-03-29T11:15:23.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-29T11:15:23.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi019.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:23.492 DEBUG:teuthology.orchestra.run.smithi019:> true 2024-03-29T11:15:23.569 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi019.front.sepia.ceph.com' 2024-03-29T11:15:23.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-29T11:15:23.570 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:15:23.661 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-03-29T11:15:23.745 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-29T11:15:23.746 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:15:23.759 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:15:23.759 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:23.759 DEBUG:teuthology.orchestra.run.smithi019:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:23.762 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:15:23.762 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:15:23.792 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:23.809 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:23.817 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:15:23.834 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:15:23.839 INFO:teuthology.orchestra.run.smithi019.stderr:sudo: ntpd: command not found 2024-03-29T11:15:23.852 INFO:teuthology.orchestra.run.smithi019.stdout:506 Cannot talk to daemon 2024-03-29T11:15:23.864 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-03-29T11:15:23.870 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:23.878 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-03-29T11:15:23.885 INFO:teuthology.orchestra.run.smithi019.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:23.895 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:15:23.911 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:15:23.954 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:23.957 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:23.957 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-29T11:15:23.957 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.957 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.958 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.958 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.964 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:15:23.968 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:15:23.968 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-03-29T11:15:23.968 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.968 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.968 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.968 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:15:23.968 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:15:23.976 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:15:23.976 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:15:23.976 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:15:23.976 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:15:23.983 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:15:23.983 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:15:23.983 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:23.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:23.986 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:15:23.986 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:15:24.131 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:24.131 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:24.219 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:15:24.219 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:15:24.263 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:24.264 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:15:24.264 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:24.293 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:24.293 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:24.344 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:15:24.344 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:15:24.344 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:15:24.362 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:24.372 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:15:24.372 DEBUG:teuthology.orchestra.run.smithi019:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:15:24.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:15:24.441 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:24.508 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-03-29T11:15:24.509 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-03-29T11:15:24.520 DEBUG:teuthology.orchestra.run.smithi019:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:15:24.587 INFO:teuthology.orchestra.run.smithi019.stdout:check_obsoletes = 1 2024-03-29T11:15:24.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum clean all 2024-03-29T11:15:24.768 INFO:teuthology.orchestra.run.smithi062.stdout:74 files removed 2024-03-29T11:15:24.795 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:24.835 INFO:teuthology.orchestra.run.smithi019.stdout:74 files removed 2024-03-29T11:15:24.863 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-radosgw 2024-03-29T11:15:25.506 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 202 kB/s | 82 kB 00:00 2024-03-29T11:15:25.526 INFO:teuthology.orchestra.run.smithi019.stdout:ceph packages for x86_64 231 kB/s | 82 kB 00:00 2024-03-29T11:15:25.881 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T11:15:25.960 INFO:teuthology.orchestra.run.smithi019.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-03-29T11:15:26.273 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2024-03-29T11:15:26.377 INFO:teuthology.orchestra.run.smithi019.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:15:26.684 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:15:26.821 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T11:15:28.187 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-29T11:15:28.505 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:15:29.444 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-03-29T11:15:31.954 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T11:15:36.221 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T11:15:37.729 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - BaseOS 927 kB/s | 8.0 MB 00:08 2024-03-29T11:15:38.064 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-29T11:15:39.657 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:15:44.840 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T11:15:47.557 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:15:47.561 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:15:47.561 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:15:47.562 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:15:47.563 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout:Installing weak dependencies: 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:15:47.564 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:Install 30 Packages 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 2 Packages 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 58 M 2024-03-29T11:15:47.565 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:15:49.450 INFO:teuthology.orchestra.run.smithi062.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 5.3 MB 00:01 2024-03-29T11:15:49.517 INFO:teuthology.orchestra.run.smithi062.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 25 kB 00:00 2024-03-29T11:15:49.742 INFO:teuthology.orchestra.run.smithi062.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.2 MB/s | 742 kB 00:00 2024-03-29T11:15:50.051 INFO:teuthology.orchestra.run.smithi062.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.1 MB/s | 9.7 MB 00:02 2024-03-29T11:15:50.211 INFO:teuthology.orchestra.run.smithi062.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.0 MB/s | 498 kB 00:00 2024-03-29T11:15:50.293 INFO:teuthology.orchestra.run.smithi062.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 553 kB/s | 45 kB 00:00 2024-03-29T11:15:50.385 INFO:teuthology.orchestra.run.smithi062.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T11:15:50.477 INFO:teuthology.orchestra.run.smithi062.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.7 MB/s | 162 kB 00:00 2024-03-29T11:15:50.635 INFO:teuthology.orchestra.run.smithi062.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 2.0 MB/s | 320 kB 00:00 2024-03-29T11:15:50.802 INFO:teuthology.orchestra.run.smithi062.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 298 kB 00:00 2024-03-29T11:15:50.936 INFO:teuthology.orchestra.run.smithi062.stdout:(11/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 5.8 MB/s | 5.1 MB 00:00 2024-03-29T11:15:50.961 INFO:teuthology.orchestra.run.smithi062.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 627 kB/s | 99 kB 00:00 2024-03-29T11:15:51.087 INFO:teuthology.orchestra.run.smithi062.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 705 kB/s | 106 kB 00:00 2024-03-29T11:15:51.112 INFO:teuthology.orchestra.run.smithi062.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-29T11:15:51.137 INFO:teuthology.orchestra.run.smithi062.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-29T11:15:51.179 INFO:teuthology.orchestra.run.smithi062.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-03-29T11:15:51.204 INFO:teuthology.orchestra.run.smithi062.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.1 MB/s | 282 kB 00:00 2024-03-29T11:15:51.229 INFO:teuthology.orchestra.run.smithi062.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T11:15:51.272 INFO:teuthology.orchestra.run.smithi062.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 220 kB 00:00 2024-03-29T11:15:51.297 INFO:teuthology.orchestra.run.smithi062.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 522 kB/s | 35 kB 00:00 2024-03-29T11:15:51.322 INFO:teuthology.orchestra.run.smithi062.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-03-29T11:15:51.556 INFO:teuthology.orchestra.run.smithi062.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-29T11:15:51.606 INFO:teuthology.orchestra.run.smithi062.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-03-29T11:15:51.665 INFO:teuthology.orchestra.run.smithi062.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-29T11:15:51.732 INFO:teuthology.orchestra.run.smithi062.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T11:15:51.757 INFO:teuthology.orchestra.run.smithi062.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 730 kB/s | 67 kB 00:00 2024-03-29T11:15:51.808 INFO:teuthology.orchestra.run.smithi062.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-29T11:15:51.833 INFO:teuthology.orchestra.run.smithi062.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T11:15:51.891 INFO:teuthology.orchestra.run.smithi062.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-29T11:15:52.208 INFO:teuthology.orchestra.run.smithi062.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 20 MB 00:04 2024-03-29T11:15:52.600 INFO:teuthology.orchestra.run.smithi062.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.4 MB/s | 3.1 MB 00:00 2024-03-29T11:15:52.692 INFO:teuthology.orchestra.run.smithi062.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.9 MB/s | 3.4 MB 00:00 2024-03-29T11:15:52.696 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:15:52.696 INFO:teuthology.orchestra.run.smithi062.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-29T11:15:52.696 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:15:52.734 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:15:52.734 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:15:53.230 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:15:53.231 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:15:54.311 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-29T11:15:54.872 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:15:54.950 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:15:54.980 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:15:55.014 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:15:55.045 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:15:55.062 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:15:55.203 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:15:55.260 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:15:55.595 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:15:55.649 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:55.742 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:15:55.859 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:15:55.884 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:15:55.930 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:15:55.959 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:15:56.003 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:15:56.073 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:15:56.103 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:15:56.150 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:15:56.185 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:15:56.297 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:15:56.337 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:15:56.377 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:15:56.410 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:15:56.438 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:15:56.471 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:15:56.800 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:15:56.826 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:15:56.900 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:15:56.931 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:15:57.022 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:15:57.505 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:15:57.537 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:15:57.592 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:15:57.663 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:15:57.697 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:15:59.997 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:00.036 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:00.170 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-29T11:16:00.619 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:00.656 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:00.741 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:00.741 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:00.741 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:00.782 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:01.821 INFO:teuthology.orchestra.run.smithi019.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-29T11:16:03.000 INFO:teuthology.orchestra.run.smithi019.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-29T11:16:08.148 INFO:teuthology.orchestra.run.smithi019.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-03-29T11:16:10.510 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-03-29T11:16:10.511 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-03-29T11:16:10.511 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:10.873 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrading: 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:10.882 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:10.883 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:10.884 INFO:teuthology.orchestra.run.smithi019.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout:Installing weak dependencies: 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout:Install 30 Packages 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout:Upgrade 2 Packages 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:10.885 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 58 M 2024-03-29T11:16:10.886 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:16:11.443 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:11.468 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:11.468 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:11.468 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:11.468 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:11.468 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:11.468 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:11.470 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:11.537 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:11.538 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:12.416 INFO:teuthology.orchestra.run.smithi019.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.9 MB/s | 5.3 MB 00:01 2024-03-29T11:16:12.483 INFO:teuthology.orchestra.run.smithi019.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 25 kB 00:00 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:12.631 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:12.632 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:12.633 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:12.634 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:12.634 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:12.634 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:12.634 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:12.650 INFO:teuthology.orchestra.run.smithi019.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.3 MB/s | 742 kB 00:00 2024-03-29T11:16:12.800 INFO:teuthology.orchestra.run.smithi019.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.2 MB/s | 498 kB 00:00 2024-03-29T11:16:12.950 INFO:teuthology.orchestra.run.smithi019.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 5.1 MB/s | 9.7 MB 00:01 2024-03-29T11:16:13.009 INFO:teuthology.orchestra.run.smithi019.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 775 kB/s | 45 kB 00:00 2024-03-29T11:16:13.067 INFO:teuthology.orchestra.run.smithi019.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T11:16:13.142 INFO:teuthology.orchestra.run.smithi019.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T11:16:13.218 INFO:teuthology.orchestra.run.smithi019.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 4.2 MB/s | 320 kB 00:00 2024-03-29T11:16:13.293 INFO:teuthology.orchestra.run.smithi019.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T11:16:13.360 INFO:teuthology.orchestra.run.smithi019.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 99 kB 00:00 2024-03-29T11:16:13.461 INFO:teuthology.orchestra.run.smithi019.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 1.0 MB/s | 106 kB 00:00 2024-03-29T11:16:13.519 INFO:teuthology.orchestra.run.smithi019.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-29T11:16:13.558 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.559 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:13.560 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:13.561 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:16:13.594 INFO:teuthology.orchestra.run.smithi019.stdout:(14/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 6.4 MB/s | 5.1 MB 00:00 2024-03-29T11:16:13.620 INFO:teuthology.orchestra.run.smithi019.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 440 kB/s | 44 kB 00:00 2024-03-29T11:16:13.686 INFO:teuthology.orchestra.run.smithi019.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-29T11:16:13.712 INFO:teuthology.orchestra.run.smithi019.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-29T11:16:13.745 INFO:teuthology.orchestra.run.smithi019.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 706 kB/s | 41 kB 00:00 2024-03-29T11:16:13.787 INFO:teuthology.orchestra.run.smithi019.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2024-03-29T11:16:13.813 INFO:teuthology.orchestra.run.smithi019.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 519 kB/s | 35 kB 00:00 2024-03-29T11:16:13.838 INFO:teuthology.orchestra.run.smithi019.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T11:16:13.879 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-test 2024-03-29T11:16:13.997 INFO:teuthology.orchestra.run.smithi019.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T11:16:14.038 INFO:teuthology.orchestra.run.smithi019.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 624 kB/s | 26 kB 00:00 2024-03-29T11:16:14.080 INFO:teuthology.orchestra.run.smithi019.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-29T11:16:14.148 INFO:teuthology.orchestra.run.smithi019.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-29T11:16:14.173 INFO:teuthology.orchestra.run.smithi019.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-29T11:16:14.206 INFO:teuthology.orchestra.run.smithi019.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-29T11:16:14.232 INFO:teuthology.orchestra.run.smithi019.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-29T11:16:14.273 INFO:teuthology.orchestra.run.smithi019.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-03-29T11:16:14.448 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:16:14.610 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:16:14.611 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:16:14.611 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:16:14.611 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:16:14.611 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2024-03-29T11:16:14.612 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:16:14.613 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 48 M 2024-03-29T11:16:14.613 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 204 M 2024-03-29T11:16:14.613 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:16:15.016 INFO:teuthology.orchestra.run.smithi019.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 4.2 MB/s | 3.1 MB 00:00 2024-03-29T11:16:15.112 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 648 kB/s | 217 kB 00:00 2024-03-29T11:16:15.216 INFO:teuthology.orchestra.run.smithi019.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.4 MB/s | 3.4 MB 00:00 2024-03-29T11:16:15.237 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.3 MB/s | 300 kB 00:00 2024-03-29T11:16:15.304 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 939 kB/s | 63 kB 00:00 2024-03-29T11:16:15.396 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 264 kB/s | 163 kB 00:00 2024-03-29T11:16:15.821 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 362 kB/s | 187 kB 00:00 2024-03-29T11:16:16.433 INFO:teuthology.orchestra.run.smithi019.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 3.7 MB/s | 20 MB 00:05 2024-03-29T11:16:16.436 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:16.437 INFO:teuthology.orchestra.run.smithi019.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-29T11:16:16.437 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:16:16.477 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:16:16.477 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:16:16.965 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:16:16.966 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:16:18.472 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:16:18.546 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:16:18.579 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:16:18.604 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:16:18.627 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:18.645 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:16:18.794 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:18.851 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:16:19.199 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:16:19.273 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:19.348 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:19.439 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:16:19.466 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:19.529 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:16:19.558 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:16:19.602 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:16:19.671 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:19.693 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:19.757 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:16:19.792 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:16:19.905 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:16:19.935 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:16:19.975 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:16:20.008 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:16:20.037 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:16:20.070 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:16:20.401 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:16:20.441 INFO:teuthology.orchestra.run.smithi019.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:20.531 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:16:20.562 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:16:20.608 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:16:21.088 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:16:21.119 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:21.190 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:16:21.260 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:16:21.287 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:16:23.501 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:23.542 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:24.138 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:16:24.179 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:24.254 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:16:24.254 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:16:24.254 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:16:24.254 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:24.289 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:34.136 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:34.136 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /sys 2024-03-29T11:16:34.136 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /proc 2024-03-29T11:16:34.136 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /mnt 2024-03-29T11:16:34.136 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /var/tmp 2024-03-29T11:16:34.136 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /home 2024-03-29T11:16:34.137 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /root 2024-03-29T11:16:34.137 INFO:teuthology.orchestra.run.smithi019.stdout:skipping the directory /tmp 2024-03-29T11:16:34.137 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:35.064 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:35.088 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:16:35.088 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:16:35.088 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:16:35.089 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:35.089 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:16:35.089 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:35.091 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:16:35.159 INFO:teuthology.orchestra.run.smithi019.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:16:36.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:16:36.250 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:16:36.251 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:16:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:16:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:37.256 INFO:teuthology.orchestra.run.smithi019.stdout:Upgraded: 2024-03-29T11:16:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.256 INFO:teuthology.orchestra.run.smithi019.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.256 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.257 INFO:teuthology.orchestra.run.smithi019.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:16:37.258 INFO:teuthology.orchestra.run.smithi019.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:16:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:16:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:16:37.259 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:37.259 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:16:37.571 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-test 2024-03-29T11:16:38.137 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:00:30 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:16:38.303 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:16:38.304 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:16:38.304 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:16:38.304 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:16:38.304 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:16:38.304 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:16:38.304 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-03-29T11:16:38.305 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:16:38.306 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 48 M 2024-03-29T11:16:38.306 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 204 M 2024-03-29T11:16:38.306 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:16:38.508 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:16:38.550 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:16:38.575 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-29T11:16:38.709 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:16:39.159 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 163 kB 00:00 2024-03-29T11:16:54.792 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 47 MB 00:40 2024-03-29T11:16:54.794 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:16:54.795 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.2 MB/s | 48 MB 00:40 2024-03-29T11:16:54.806 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:16:54.827 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:16:54.827 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:16:55.228 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:16:55.228 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:16:55.820 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:16:55.903 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:16:55.942 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:16:55.975 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:16:56.029 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:16:56.054 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:00.774 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:00.848 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:03.248 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:03.249 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:03.249 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:03.249 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:03.249 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:03.249 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:03.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:03.620 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:03.620 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:03.620 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:03.621 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:03.963 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph 2024-03-29T11:17:04.530 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:04.689 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout:========================================================================================= 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout:========================================================================================= 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:04.692 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:04.693 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:04.694 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:04.695 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:04.696 INFO:teuthology.orchestra.run.smithi062.stdout:========================================================================================= 2024-03-29T11:17:04.696 INFO:teuthology.orchestra.run.smithi062.stdout:Install 36 Packages 2024-03-29T11:17:04.696 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:04.696 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 30 M 2024-03-29T11:17:04.696 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 107 M 2024-03-29T11:17:04.696 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:05.374 INFO:teuthology.orchestra.run.smithi062.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 23 kB/s | 6.6 kB 00:00 2024-03-29T11:17:06.124 INFO:teuthology.orchestra.run.smithi062.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2024-03-29T11:17:06.333 INFO:teuthology.orchestra.run.smithi062.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 2.1 MB 00:01 2024-03-29T11:17:06.441 INFO:teuthology.orchestra.run.smithi062.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.2 MB/s | 247 kB 00:00 2024-03-29T11:17:06.558 INFO:teuthology.orchestra.run.smithi062.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-29T11:17:06.600 INFO:teuthology.orchestra.run.smithi062.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:06.634 INFO:teuthology.orchestra.run.smithi062.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-29T11:17:06.667 INFO:teuthology.orchestra.run.smithi062.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:17:06.701 INFO:teuthology.orchestra.run.smithi062.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:17:06.743 INFO:teuthology.orchestra.run.smithi062.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:17:06.810 INFO:teuthology.orchestra.run.smithi062.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:17:06.843 INFO:teuthology.orchestra.run.smithi062.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:17:06.877 INFO:teuthology.orchestra.run.smithi062.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-29T11:17:06.919 INFO:teuthology.orchestra.run.smithi062.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:17:07.053 INFO:teuthology.orchestra.run.smithi062.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-29T11:17:07.195 INFO:teuthology.orchestra.run.smithi062.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 158 kB/s | 22 kB 00:00 2024-03-29T11:17:07.462 INFO:teuthology.orchestra.run.smithi062.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-29T11:17:07.487 INFO:teuthology.orchestra.run.smithi062.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 517 kB/s | 151 kB 00:00 2024-03-29T11:17:07.604 INFO:teuthology.orchestra.run.smithi062.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-03-29T11:17:07.671 INFO:teuthology.orchestra.run.smithi062.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 208 kB/s | 43 kB 00:00 2024-03-29T11:17:07.721 INFO:teuthology.orchestra.run.smithi062.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-29T11:17:07.746 INFO:teuthology.orchestra.run.smithi062.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-03-29T11:17:07.788 INFO:teuthology.orchestra.run.smithi062.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 264 kB/s | 18 kB 00:00 2024-03-29T11:17:07.813 INFO:teuthology.orchestra.run.smithi062.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-29T11:17:07.856 INFO:teuthology.orchestra.run.smithi062.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-03-29T11:17:07.881 INFO:teuthology.orchestra.run.smithi062.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-03-29T11:17:07.923 INFO:teuthology.orchestra.run.smithi062.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 693 kB/s | 46 kB 00:00 2024-03-29T11:17:07.998 INFO:teuthology.orchestra.run.smithi062.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-29T11:17:08.024 INFO:teuthology.orchestra.run.smithi062.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 554 kB/s | 79 kB 00:00 2024-03-29T11:17:08.066 INFO:teuthology.orchestra.run.smithi062.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 245 kB/s | 16 kB 00:00 2024-03-29T11:17:08.099 INFO:teuthology.orchestra.run.smithi062.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T11:17:08.133 INFO:teuthology.orchestra.run.smithi062.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-03-29T11:17:08.217 INFO:teuthology.orchestra.run.smithi062.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T11:17:08.242 INFO:teuthology.orchestra.run.smithi062.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-29T11:17:08.300 INFO:teuthology.orchestra.run.smithi062.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 239 kB/s | 20 kB 00:00 2024-03-29T11:17:10.709 INFO:teuthology.orchestra.run.smithi062.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 3.7 MB/s | 17 MB 00:04 2024-03-29T11:17:10.712 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:10.712 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.9 MB/s | 30 MB 00:06 2024-03-29T11:17:10.738 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:10.785 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:10.785 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:11.236 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:11.236 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:12.716 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:12.778 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:12.821 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:12.856 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:12.930 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:12.971 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:13.004 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:13.102 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:13.177 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:13.224 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:13.298 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:13.351 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:13.396 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:13.455 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:13.519 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:13.713 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:13.767 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:13.814 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:13.848 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:13.879 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:13.957 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:13.997 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:14.030 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:14.072 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:14.118 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:14.149 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:14.239 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 47 MB 00:35 2024-03-29T11:17:14.242 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:14.242 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:14.243 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.3 MB/s | 48 MB 00:35 2024-03-29T11:17:14.257 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:17:14.280 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:17:14.280 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:17:14.280 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:14.363 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:14.514 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:14.643 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:14.675 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:17:14.675 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:17:14.812 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:14.843 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:14.868 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:14.868 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:14.869 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:14.869 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:14.869 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:14.869 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:15.265 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:17:15.341 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:15.414 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:15.454 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:15.506 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:15.534 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:16.385 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:16.411 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:16.411 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:16.411 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:16.411 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:16.411 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:16.869 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:16.892 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:16.892 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:16.892 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:16.892 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:16.892 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:16.892 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:17.112 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:17.135 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:17.135 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:17.135 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:17.135 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:17.135 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:17.135 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:17.204 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:18.867 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:18.868 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:18.869 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:18.870 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:20.016 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:20.017 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:20.018 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:20.019 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:20.019 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:20.019 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:20.019 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:20.019 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:20.249 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:20.328 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:20.360 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-base 2024-03-29T11:17:20.965 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:21.066 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:21.121 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:21.122 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:17:21.123 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:21.183 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install cephadm 2024-03-29T11:17:21.757 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:21.919 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:21.919 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:17:21.919 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 747 k 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 754 k 2024-03-29T11:17:21.920 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:22.361 INFO:teuthology.orchestra.run.smithi062.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T11:17:22.362 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:22.362 INFO:teuthology.orchestra.run.smithi062.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T11:17:22.362 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:22.367 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:22.368 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:22.373 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:22.374 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:22.447 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:22.702 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:17:22.702 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:17:22.703 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:17:22.703 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:22.703 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:17:22.703 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:17:23.036 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:23.106 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:23.107 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:23.133 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:23.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph 2024-03-29T11:17:23.526 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:23.851 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:23.851 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:23.851 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:23.851 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:23.851 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:23.851 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:24.022 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:16 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:24.070 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:24.181 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:17:24.184 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:17:24.185 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:17:24.186 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:17:24.187 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:17:24.187 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:17:24.187 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:17:24.187 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:24.188 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi019.stdout:========================================================================================= 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi019.stdout:Install 36 Packages 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 30 M 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 107 M 2024-03-29T11:17:24.189 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:17:24.644 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:24.800 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:24.800 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:17:24.801 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 145 k 2024-03-29T11:17:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 436 k 2024-03-29T11:17:24.802 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:25.085 INFO:teuthology.orchestra.run.smithi062.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 516 kB/s | 145 kB 00:00 2024-03-29T11:17:25.085 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:25.085 INFO:teuthology.orchestra.run.smithi062.stdout:Total 514 kB/s | 145 kB 00:00 2024-03-29T11:17:25.085 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:25.092 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:25.093 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:25.126 INFO:teuthology.orchestra.run.smithi019.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-03-29T11:17:25.149 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:25.149 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:25.290 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:25.343 INFO:teuthology.orchestra.run.smithi019.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.8 MB/s | 1.5 MB 00:00 2024-03-29T11:17:25.368 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:25.393 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:25.393 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:25.393 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:25.393 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:25.401 INFO:teuthology.orchestra.run.smithi019.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 2.1 MB 00:00 2024-03-29T11:17:25.496 INFO:teuthology.orchestra.run.smithi019.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.6 MB/s | 247 kB 00:00 2024-03-29T11:17:25.653 INFO:teuthology.orchestra.run.smithi019.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:17:25.694 INFO:teuthology.orchestra.run.smithi019.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:17:25.728 INFO:teuthology.orchestra.run.smithi019.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-03-29T11:17:25.770 INFO:teuthology.orchestra.run.smithi019.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-29T11:17:25.803 INFO:teuthology.orchestra.run.smithi019.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:17:25.853 INFO:teuthology.orchestra.run.smithi019.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:17:25.979 INFO:teuthology.orchestra.run.smithi019.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 9.2 MB/s | 1.1 MB 00:00 2024-03-29T11:17:26.037 INFO:teuthology.orchestra.run.smithi019.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-03-29T11:17:26.096 INFO:teuthology.orchestra.run.smithi019.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-29T11:17:26.146 INFO:teuthology.orchestra.run.smithi019.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:17:26.362 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:26.362 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:26.363 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:26.363 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:26.363 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:26.363 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:26.605 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr 2024-03-29T11:17:26.663 INFO:teuthology.orchestra.run.smithi019.stdout:(15/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-29T11:17:26.688 INFO:teuthology.orchestra.run.smithi019.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 230 kB/s | 125 kB 00:00 2024-03-29T11:17:26.789 INFO:teuthology.orchestra.run.smithi019.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 179 kB/s | 22 kB 00:00 2024-03-29T11:17:26.814 INFO:teuthology.orchestra.run.smithi019.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-29T11:17:26.864 INFO:teuthology.orchestra.run.smithi019.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-03-29T11:17:26.889 INFO:teuthology.orchestra.run.smithi019.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-03-29T11:17:26.957 INFO:teuthology.orchestra.run.smithi019.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:17:26.982 INFO:teuthology.orchestra.run.smithi019.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-29T11:17:27.007 INFO:teuthology.orchestra.run.smithi019.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-29T11:17:27.041 INFO:teuthology.orchestra.run.smithi019.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-03-29T11:17:27.066 INFO:teuthology.orchestra.run.smithi019.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2024-03-29T11:17:27.083 INFO:teuthology.orchestra.run.smithi019.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 470 kB/s | 20 kB 00:00 2024-03-29T11:17:27.100 INFO:teuthology.orchestra.run.smithi019.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-03-29T11:17:27.142 INFO:teuthology.orchestra.run.smithi019.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T11:17:27.167 INFO:teuthology.orchestra.run.smithi019.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:17:27.181 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:27.184 INFO:teuthology.orchestra.run.smithi019.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-03-29T11:17:27.210 INFO:teuthology.orchestra.run.smithi019.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T11:17:27.226 INFO:teuthology.orchestra.run.smithi019.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 854 kB/s | 36 kB 00:00 2024-03-29T11:17:27.251 INFO:teuthology.orchestra.run.smithi019.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-03-29T11:17:27.283 INFO:teuthology.orchestra.run.smithi062.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:27.285 INFO:teuthology.orchestra.run.smithi019.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 601 kB/s | 20 kB 00:00 2024-03-29T11:17:27.338 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:27.339 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:17:27.340 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:27.352 INFO:teuthology.orchestra.run.smithi019.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-03-29T11:17:27.402 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:27.971 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:28.130 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout:====================================================================================== 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout:====================================================================================== 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:28.131 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout:====================================================================================== 2024-03-29T11:17:28.132 INFO:teuthology.orchestra.run.smithi062.stdout:Install 12 Packages 2024-03-29T11:17:28.133 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:28.133 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 6.9 M 2024-03-29T11:17:28.133 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 26 M 2024-03-29T11:17:28.133 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:28.962 INFO:teuthology.orchestra.run.smithi062.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 99 kB/s | 31 kB 00:00 2024-03-29T11:17:29.021 INFO:teuthology.orchestra.run.smithi062.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 41 kB/s | 15 kB 00:00 2024-03-29T11:17:29.338 INFO:teuthology.orchestra.run.smithi062.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-03-29T11:17:29.363 INFO:teuthology.orchestra.run.smithi062.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 698 kB/s | 238 kB 00:00 2024-03-29T11:17:29.513 INFO:teuthology.orchestra.run.smithi062.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-29T11:17:29.639 INFO:teuthology.orchestra.run.smithi062.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 156 kB/s | 19 kB 00:00 2024-03-29T11:17:29.664 INFO:teuthology.orchestra.run.smithi062.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T11:17:29.722 INFO:teuthology.orchestra.run.smithi062.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-03-29T11:17:29.781 INFO:teuthology.orchestra.run.smithi062.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 526 kB/s | 31 kB 00:00 2024-03-29T11:17:29.848 INFO:teuthology.orchestra.run.smithi062.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T11:17:30.485 INFO:teuthology.orchestra.run.smithi019.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 895 kB/s | 4.7 MB 00:05 2024-03-29T11:17:30.489 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:30.489 INFO:teuthology.orchestra.run.smithi019.stdout:Total 4.7 MB/s | 30 MB 00:06 2024-03-29T11:17:30.515 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:17:30.560 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:17:30.560 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:17:30.573 INFO:teuthology.orchestra.run.smithi062.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:17:30.723 INFO:teuthology.orchestra.run.smithi062.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 878 kB/s | 1.8 MB 00:02 2024-03-29T11:17:30.726 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:30.729 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.6 MB/s | 6.9 MB 00:02 2024-03-29T11:17:30.746 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:30.764 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:30.765 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:30.954 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:30.955 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:30.993 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:17:30.993 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:17:31.484 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:31.566 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:31.610 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:31.657 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:31.686 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:31.789 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:31.916 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:32.008 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:32.058 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:32.111 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:32.143 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:32.375 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:17:32.399 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:32.426 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:32.429 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:17:32.465 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:17:32.499 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:17:32.573 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:17:32.614 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:17:32.647 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:17:32.742 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:17:32.820 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:17:32.867 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:17:32.941 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:17:32.994 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:17:33.031 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:17:33.089 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:17:33.152 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:33.179 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:33.180 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:33.180 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:33.180 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:33.180 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:33.344 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:17:33.393 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:17:33.431 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:17:33.466 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:17:33.496 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:17:33.575 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:17:33.614 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:33.648 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:33.649 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:33.650 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:33.650 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:33.650 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:33.650 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:33.680 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:17:33.723 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:17:33.758 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:17:33.849 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:17:33.888 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:17:33.932 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:33.964 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:17:34.113 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:17:34.235 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:34.404 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:17:34.427 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:34.452 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:17:34.452 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:34.452 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:17:34.452 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:34.452 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:17:34.452 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:34.512 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:34.671 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:34.672 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================================ 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================================ 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:34.673 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================================ 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout:Install 12 Packages 2024-03-29T11:17:34.674 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:34.675 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 37 M 2024-03-29T11:17:34.675 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 209 M 2024-03-29T11:17:34.675 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:35.031 INFO:teuthology.orchestra.run.smithi062.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 174 kB/s | 32 kB 00:00 2024-03-29T11:17:35.064 INFO:teuthology.orchestra.run.smithi062.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-03-29T11:17:35.097 INFO:teuthology.orchestra.run.smithi062.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:17:35.198 INFO:teuthology.orchestra.run.smithi062.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-03-29T11:17:35.961 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:35.986 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:17:35.986 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:35.987 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:17:35.987 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:35.987 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:17:35.987 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:36.248 INFO:teuthology.orchestra.run.smithi062.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.6 MB/s | 4.8 MB 00:01 2024-03-29T11:17:36.315 INFO:teuthology.orchestra.run.smithi062.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.9 MB/s | 4.7 MB 00:01 2024-03-29T11:17:36.340 INFO:teuthology.orchestra.run.smithi062.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:17:36.445 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:36.468 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:17:36.468 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:36.468 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:17:36.468 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:36.468 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:17:36.468 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:36.687 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:36.710 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:17:36.710 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:36.711 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:17:36.711 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:36.711 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:17:36.711 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:36.779 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:36.891 INFO:teuthology.orchestra.run.smithi062.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-03-29T11:17:37.808 INFO:teuthology.orchestra.run.smithi062.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 213 kB/s | 195 kB 00:00 2024-03-29T11:17:38.025 INFO:teuthology.orchestra.run.smithi062.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T11:17:38.217 INFO:teuthology.orchestra.run.smithi062.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 428 kB/s | 803 kB 00:01 2024-03-29T11:17:38.468 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:17:38.468 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:17:38.469 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:17:38.470 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:17:38.471 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:17:38.471 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:17:38.471 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:17:38.472 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:17:38.473 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:17:39.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:17:39.249 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:39.249 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:17:39.249 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:17:39.250 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:17:39.251 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:39.252 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:39.586 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-base 2024-03-29T11:17:40.161 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:40.263 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:40.321 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:40.322 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:17:40.322 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:40.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install cephadm 2024-03-29T11:17:40.976 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:41.134 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:41.134 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:17:41.134 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 747 k 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 754 k 2024-03-29T11:17:41.135 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:17:42.469 INFO:teuthology.orchestra.run.smithi019.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 561 kB/s | 747 kB 00:01 2024-03-29T11:17:42.469 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:42.470 INFO:teuthology.orchestra.run.smithi019.stdout:Total 560 kB/s | 747 kB 00:01 2024-03-29T11:17:42.470 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:17:42.477 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:17:42.477 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:17:42.483 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:17:42.484 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:17:42.569 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:17:42.997 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:43.093 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:43.366 INFO:teuthology.orchestra.run.smithi062.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 888 kB/s | 7.4 MB 00:08 2024-03-29T11:17:43.370 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:43.370 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.3 MB/s | 37 MB 00:08 2024-03-29T11:17:43.414 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:43.440 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:43.441 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:43.460 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:43.750 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:17:43.750 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:43.750 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:17:43.750 INFO:teuthology.orchestra.run.smithi019.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:43.750 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:43.751 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:43.808 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:43.808 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:43.973 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:17:44.548 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:44.653 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:44.703 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:44.704 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:17:44.705 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:17:44.705 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:17:44.705 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:44.705 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 145 k 2024-03-29T11:17:44.705 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 436 k 2024-03-29T11:17:44.705 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:17:44.732 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:44.763 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:44.863 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:45.016 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:45.033 INFO:teuthology.orchestra.run.smithi019.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 444 kB/s | 145 kB 00:00 2024-03-29T11:17:45.034 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:45.034 INFO:teuthology.orchestra.run.smithi019.stdout:Total 441 kB/s | 145 kB 00:00 2024-03-29T11:17:45.034 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:17:45.042 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:17:45.042 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:17:45.097 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:17:45.097 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:17:45.200 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:45.231 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:45.242 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:17:45.301 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:45.310 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:45.325 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:45.326 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:45.326 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:17:45.326 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:45.998 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:46.115 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:46.249 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:17:46.249 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:46.250 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:17:46.250 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:46.250 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:46.250 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:46.511 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr 2024-03-29T11:17:47.088 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:47.191 INFO:teuthology.orchestra.run.smithi019.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:17:47.246 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:47.247 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:17:47.247 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:47.333 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:17:47.453 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:17:47.905 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:48.062 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:48.063 INFO:teuthology.orchestra.run.smithi019.stdout:====================================================================================== 2024-03-29T11:17:48.063 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:17:48.063 INFO:teuthology.orchestra.run.smithi019.stdout:====================================================================================== 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:17:48.064 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout:====================================================================================== 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.9 M 2024-03-29T11:17:48.065 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-03-29T11:17:48.066 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:17:48.531 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:17:48.564 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:48.779 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 88 kB/s | 31 kB 00:00 2024-03-29T11:17:48.804 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 40 kB/s | 15 kB 00:00 2024-03-29T11:17:48.946 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:17:48.979 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:17:49.163 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-03-29T11:17:49.272 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T11:17:49.288 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-29T11:17:49.431 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-03-29T11:17:49.447 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 911 kB/s | 144 kB 00:00 2024-03-29T11:17:49.498 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 460 kB/s | 31 kB 00:00 2024-03-29T11:17:49.556 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-29T11:17:49.890 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.2 MB/s | 1.8 MB 00:01 2024-03-29T11:17:49.893 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:49.893 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.7 MB/s | 6.9 MB 00:01 2024-03-29T11:17:49.913 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:17:49.930 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:17:49.930 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:17:50.112 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:17:50.112 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:17:50.637 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:17:50.716 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:17:50.759 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:17:50.806 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:17:50.836 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:17:50.943 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:17:50.963 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:17:50.964 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:17:51.075 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:17:51.176 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:17:51.224 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:17:51.252 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:51.284 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:17:51.377 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:17:51.378 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:51.379 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:17:51.379 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:17:51.379 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:51.379 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:51.539 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:17:51.567 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:51.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:17:52.307 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:08 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:52.327 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:17:52.328 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:17:52.465 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:17:52.467 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout:Install 13 Packages 2024-03-29T11:17:52.468 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:52.469 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 2.0 M 2024-03-29T11:17:52.469 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 26 M 2024-03-29T11:17:52.469 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:52.655 INFO:teuthology.orchestra.run.smithi062.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 222 kB/s | 24 kB 00:00 2024-03-29T11:17:52.681 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:17:52.681 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:52.681 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:17:52.682 INFO:teuthology.orchestra.run.smithi019.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:17:52.683 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:52.683 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:17:52.705 INFO:teuthology.orchestra.run.smithi062.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 113 kB/s | 18 kB 00:00 2024-03-29T11:17:52.747 INFO:teuthology.orchestra.run.smithi062.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-29T11:17:52.772 INFO:teuthology.orchestra.run.smithi062.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:17:52.797 INFO:teuthology.orchestra.run.smithi062.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:17:52.823 INFO:teuthology.orchestra.run.smithi062.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 870 kB/s | 43 kB 00:00 2024-03-29T11:17:52.940 INFO:teuthology.orchestra.run.smithi062.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T11:17:52.957 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:17:52.965 INFO:teuthology.orchestra.run.smithi062.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T11:17:53.123 INFO:teuthology.orchestra.run.smithi062.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.1 MB/s | 203 kB 00:00 2024-03-29T11:17:53.182 INFO:teuthology.orchestra.run.smithi062.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-29T11:17:53.241 INFO:teuthology.orchestra.run.smithi062.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-29T11:17:53.266 INFO:teuthology.orchestra.run.smithi062.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-29T11:17:53.350 INFO:teuthology.orchestra.run.smithi062.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 65 kB/s | 52 kB 00:00 2024-03-29T11:17:53.351 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:53.352 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-03-29T11:17:53.352 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:53.366 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:53.366 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:53.463 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:53.465 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:53.541 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:17:53.701 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:17:53.702 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-29T11:17:53.702 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:17:53.702 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-29T11:17:53.702 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:17:53.703 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout:============================================================================================ 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout:Install 12 Packages 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 37 M 2024-03-29T11:17:53.704 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 209 M 2024-03-29T11:17:53.705 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:17:53.977 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:54.155 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:17:54.171 INFO:teuthology.orchestra.run.smithi019.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 245 kB/s | 32 kB 00:00 2024-03-29T11:17:54.205 INFO:teuthology.orchestra.run.smithi019.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 531 kB/s | 17 kB 00:00 2024-03-29T11:17:54.218 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:17:54.280 INFO:teuthology.orchestra.run.smithi019.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:17:54.285 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:17:54.306 INFO:teuthology.orchestra.run.smithi019.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 348 kB/s | 34 kB 00:00 2024-03-29T11:17:54.319 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:17:54.361 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:17:54.459 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:17:54.481 INFO:teuthology.orchestra.run.smithi019.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2024-03-29T11:17:54.548 INFO:teuthology.orchestra.run.smithi019.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T11:17:54.565 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:17:54.573 INFO:teuthology.orchestra.run.smithi019.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:17:54.620 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:17:54.882 INFO:teuthology.orchestra.run.smithi019.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-29T11:17:54.916 INFO:teuthology.orchestra.run.smithi019.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-29T11:17:54.991 INFO:teuthology.orchestra.run.smithi019.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 195 kB 00:00 2024-03-29T11:17:55.074 INFO:teuthology.orchestra.run.smithi019.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-29T11:17:55.387 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:17:55.441 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:17:55.475 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:17:55.525 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:17:55.553 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:56.016 INFO:teuthology.orchestra.run.smithi019.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.7 MB/s | 7.4 MB 00:01 2024-03-29T11:17:56.019 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:56.019 INFO:teuthology.orchestra.run.smithi019.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-29T11:17:56.062 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:17:56.088 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:17:56.088 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:17:56.315 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:17:56.316 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:17:56.316 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:17:56.316 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:17:56.316 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:17:56.316 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:17:56.316 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:17:56.455 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:17:56.455 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:17:56.678 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:17:56.679 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:17:56.680 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:17:56.680 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:17:56.680 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:56.680 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:17:56.914 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:17:57.171 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:17:57.231 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:17:57.262 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:17:57.360 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:17:57.407 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:17:57.492 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:13 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:17:57.591 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:17:57.614 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:17:57.651 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:17:57.652 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout:======================================================================================== 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout:Install 6 Packages 2024-03-29T11:17:57.653 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:17:57.654 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 6.8 M 2024-03-29T11:17:57.654 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 33 M 2024-03-29T11:17:57.654 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:17:57.706 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:17:58.401 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:17:58.513 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:17:58.576 INFO:teuthology.orchestra.run.smithi062.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 278 kB/s | 138 kB 00:00 2024-03-29T11:17:58.876 INFO:teuthology.orchestra.run.smithi062.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-29T11:17:58.902 INFO:teuthology.orchestra.run.smithi062.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 302 kB/s | 248 kB 00:00 2024-03-29T11:17:58.952 INFO:teuthology.orchestra.run.smithi062.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 774 kB/s | 58 kB 00:00 2024-03-29T11:17:59.060 INFO:teuthology.orchestra.run.smithi062.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.9 MB/s | 5.8 MB 00:00 2024-03-29T11:17:59.085 INFO:teuthology.orchestra.run.smithi062.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 278 kB/s | 51 kB 00:00 2024-03-29T11:17:59.087 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:59.087 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-29T11:17:59.102 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:17:59.115 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:17:59.115 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:17:59.211 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:17:59.211 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:17:59.523 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:17:59.712 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:17:59.772 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:17:59.831 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:00.322 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:00.418 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:00.487 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:00.508 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:00.907 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:00.938 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:01.335 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:01.628 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:01.629 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:01.629 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:01.876 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-fuse 2024-03-29T11:18:02.448 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:02.605 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:02.605 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:02.605 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:18:02.605 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:02.605 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout:Install 2 Packages 2024-03-29T11:18:02.606 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:02.607 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 964 k 2024-03-29T11:18:02.607 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 2.8 M 2024-03-29T11:18:02.607 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:02.729 INFO:teuthology.orchestra.run.smithi062.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 644 kB/s | 79 kB 00:00 2024-03-29T11:18:02.954 INFO:teuthology.orchestra.run.smithi062.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.5 MB/s | 885 kB 00:00 2024-03-29T11:18:02.955 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:02.956 INFO:teuthology.orchestra.run.smithi062.stdout:Total 2.7 MB/s | 964 kB 00:00 2024-03-29T11:18:02.956 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:02.978 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:02.978 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:03.054 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:03.055 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:03.246 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:03.385 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:03.434 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:03.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:03.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:03.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:03.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:03.435 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:03.458 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:03.900 INFO:teuthology.orchestra.run.smithi019.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:03.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:03.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:03.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:03.901 INFO:teuthology.orchestra.run.smithi019.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:03.901 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:03.901 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:04.186 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:04.186 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:04.226 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:04.501 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:04.501 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:04.502 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:04.502 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:04.502 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:04.502 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:04.760 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-volume 2024-03-29T11:18:04.804 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:04.962 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:04.963 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-29T11:18:04.963 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:18:04.963 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-29T11:18:04.963 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:04.964 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:04.965 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:04.965 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:04.965 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-29T11:18:04.965 INFO:teuthology.orchestra.run.smithi019.stdout:Install 13 Packages 2024-03-29T11:18:04.965 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:04.965 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 2.0 M 2024-03-29T11:18:04.966 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 26 M 2024-03-29T11:18:04.966 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:05.197 INFO:teuthology.orchestra.run.smithi019.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 140 kB/s | 18 kB 00:00 2024-03-29T11:18:05.214 INFO:teuthology.orchestra.run.smithi019.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 24 kB 00:00 2024-03-29T11:18:05.264 INFO:teuthology.orchestra.run.smithi019.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:18:05.289 INFO:teuthology.orchestra.run.smithi019.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:18:05.314 INFO:teuthology.orchestra.run.smithi019.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:18:05.331 INFO:teuthology.orchestra.run.smithi019.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T11:18:05.344 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:05.423 INFO:teuthology.orchestra.run.smithi019.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 147 kB/s | 52 kB 00:00 2024-03-29T11:18:05.465 INFO:teuthology.orchestra.run.smithi019.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 214 kB/s | 32 kB 00:00 2024-03-29T11:18:05.482 INFO:teuthology.orchestra.run.smithi019.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 99 kB/s | 15 kB 00:00 2024-03-29T11:18:05.503 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:05.503 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:05.503 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-29T11:18:05.503 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout:Install 2 Packages 2024-03-29T11:18:05.504 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:05.505 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 332 k 2024-03-29T11:18:05.505 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 1.5 M 2024-03-29T11:18:05.505 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:05.532 INFO:teuthology.orchestra.run.smithi019.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.8 MB/s | 203 kB 00:00 2024-03-29T11:18:05.549 INFO:teuthology.orchestra.run.smithi019.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-03-29T11:18:05.583 INFO:teuthology.orchestra.run.smithi019.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-29T11:18:05.617 INFO:teuthology.orchestra.run.smithi019.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-03-29T11:18:05.620 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:05.620 INFO:teuthology.orchestra.run.smithi019.stdout:Total 3.1 MB/s | 2.0 MB 00:00 2024-03-29T11:18:05.621 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:05.634 INFO:teuthology.orchestra.run.smithi062.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 531 kB/s | 69 kB 00:00 2024-03-29T11:18:05.635 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:05.635 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:05.730 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:05.730 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:06.237 INFO:teuthology.orchestra.run.smithi062.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 360 kB/s | 264 kB 00:00 2024-03-29T11:18:06.238 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:06.239 INFO:teuthology.orchestra.run.smithi062.stdout:Total 452 kB/s | 332 kB 00:00 2024-03-29T11:18:06.239 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:06.249 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:06.249 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:06.265 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:06.293 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:06.293 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:06.447 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:06.449 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:06.496 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:06.559 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:06.597 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:06.638 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:06.644 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:06.678 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:06.695 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:06.695 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:06.695 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:06.696 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:06.734 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:06.843 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:06.898 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:07.449 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:07.660 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:07.712 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:07.745 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:07.775 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:07.775 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:07.775 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:07.775 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:07.776 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:07.776 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:07.776 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:07.794 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:07.826 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:08.139 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados-devel 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:08.640 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:08.641 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:08.719 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:08.876 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:08.876 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 127 k 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 456 k 2024-03-29T11:18:08.877 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:09.006 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:09.006 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:09.007 INFO:teuthology.orchestra.run.smithi019.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:09.008 INFO:teuthology.orchestra.run.smithi019.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:09.008 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:09.008 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:09.272 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:09.434 INFO:teuthology.orchestra.run.smithi062.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 228 kB/s | 127 kB 00:00 2024-03-29T11:18:09.435 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:09.435 INFO:teuthology.orchestra.run.smithi062.stdout:Total 227 kB/s | 127 kB 00:00 2024-03-29T11:18:09.435 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:09.441 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:09.441 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:09.500 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:09.501 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:09.649 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:09.760 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:09.853 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:01 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:10.011 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:10.012 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout:======================================================================================== 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout:Install 6 Packages 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:10.013 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 6.8 M 2024-03-29T11:18:10.014 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 33 M 2024-03-29T11:18:10.014 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:10.203 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.459 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:10.459 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:10.459 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:10.459 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:10.459 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:10.460 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:10.513 INFO:teuthology.orchestra.run.smithi019.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-29T11:18:10.737 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs2 2024-03-29T11:18:10.746 INFO:teuthology.orchestra.run.smithi019.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:18:10.838 INFO:teuthology.orchestra.run.smithi019.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 268 kB/s | 138 kB 00:00 2024-03-29T11:18:11.022 INFO:teuthology.orchestra.run.smithi019.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 210 kB/s | 58 kB 00:00 2024-03-29T11:18:11.080 INFO:teuthology.orchestra.run.smithi019.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 967 kB/s | 548 kB 00:00 2024-03-29T11:18:11.105 INFO:teuthology.orchestra.run.smithi019.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 191 kB/s | 51 kB 00:00 2024-03-29T11:18:11.109 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:11.109 INFO:teuthology.orchestra.run.smithi019.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2024-03-29T11:18:11.122 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:11.133 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:11.134 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:11.227 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:11.227 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:11.319 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:11.422 INFO:teuthology.orchestra.run.smithi062.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:11.478 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:11.479 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:11.480 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:11.533 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:11.545 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:11.694 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:11.741 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:12.122 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:12.281 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 31 k 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 140 k 2024-03-29T11:18:12.282 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:12.297 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:12.387 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:12.465 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:12.494 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:12.662 INFO:teuthology.orchestra.run.smithi062.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 82 kB/s | 31 kB 00:00 2024-03-29T11:18:12.663 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:12.663 INFO:teuthology.orchestra.run.smithi062.stdout:Total 82 kB/s | 31 kB 00:00 2024-03-29T11:18:12.663 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:12.667 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:12.667 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:12.707 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:12.707 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:12.844 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:12.963 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:13.333 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:13.362 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:13.362 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:13.362 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:13.362 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:13.363 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:13.363 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:13.602 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:13.602 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:13.602 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:13.602 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:13.602 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:13.602 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:13.672 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:13.672 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:13.672 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:13.672 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:13.672 INFO:teuthology.orchestra.run.smithi019.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:13.672 INFO:teuthology.orchestra.run.smithi019.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi019.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi019.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi019.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:13.673 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:13.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librados2 2024-03-29T11:18:13.933 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-fuse 2024-03-29T11:18:14.396 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:14.495 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:14.507 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:06 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:14.549 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:14.551 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:14.551 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:14.606 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install librbd1 2024-03-29T11:18:14.664 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:14.665 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:14.665 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:18:14.665 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:14.665 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:14.665 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 964 k 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 2.8 M 2024-03-29T11:18:14.666 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:14.807 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 557 kB/s | 79 kB 00:00 2024-03-29T11:18:15.179 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:15.280 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:15.334 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:15.335 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:15.335 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:15.349 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 885 kB 00:00 2024-03-29T11:18:15.350 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:15.351 INFO:teuthology.orchestra.run.smithi019.stdout:Total 1.4 MB/s | 964 kB 00:00 2024-03-29T11:18:15.351 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:15.375 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:15.375 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:15.379 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rados 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:15.455 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:15.642 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:15.796 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:15.870 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:15.957 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:16.059 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:16.114 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:16.114 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:16.115 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:16.159 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rgw 2024-03-29T11:18:16.565 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:16.565 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:16.737 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:16.840 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:16.853 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:16.854 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:16.854 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:16.854 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:16.854 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:16.854 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:16.895 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:16.896 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:16.897 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:16.939 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-cephfs 2024-03-29T11:18:17.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install ceph-volume 2024-03-29T11:18:17.514 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:17.616 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:17.671 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:17.672 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:17.672 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:17.718 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install python3-rbd 2024-03-29T11:18:17.763 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:17.921 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repository Size 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:Installing dependencies: 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:17.922 INFO:teuthology.orchestra.run.smithi019.stdout:Install 2 Packages 2024-03-29T11:18:17.923 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:17.923 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 332 k 2024-03-29T11:18:17.923 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 1.5 M 2024-03-29T11:18:17.923 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:18.054 INFO:teuthology.orchestra.run.smithi019.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 525 kB/s | 69 kB 00:00 2024-03-29T11:18:18.293 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:34 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:18.304 INFO:teuthology.orchestra.run.smithi019.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 692 kB/s | 264 kB 00:00 2024-03-29T11:18:18.306 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:18.306 INFO:teuthology.orchestra.run.smithi019.stdout:Total 866 kB/s | 332 kB 00:00 2024-03-29T11:18:18.306 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:18.315 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:18.315 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:18.355 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:18.355 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:18.395 INFO:teuthology.orchestra.run.smithi062.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:18.450 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:18.451 INFO:teuthology.orchestra.run.smithi062.stdout:Nothing to do. 2024-03-29T11:18:18.451 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:18.506 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:18.507 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-fuse 2024-03-29T11:18:18.692 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:18.723 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:18.739 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:18.739 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:18.739 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:18.739 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:19.079 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:19.234 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:19.235 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:18:19.236 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:19.236 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 85 k 2024-03-29T11:18:19.236 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 231 k 2024-03-29T11:18:19.236 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:19.517 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:19.540 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 280 kB/s | 85 kB 00:00 2024-03-29T11:18:19.541 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:19.541 INFO:teuthology.orchestra.run.smithi062.stdout:Total 278 kB/s | 85 kB 00:00 2024-03-29T11:18:19.542 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:19.549 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:19.549 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:19.607 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:19.607 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:19.749 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:19.853 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:19.859 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:20.187 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados-devel 2024-03-29T11:18:20.234 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:20.487 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:20.709 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-mirror 2024-03-29T11:18:20.766 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:20.924 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:20.925 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:20.925 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:18:20.925 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:20.925 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 127 k 2024-03-29T11:18:20.925 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 456 k 2024-03-29T11:18:20.925 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 630 kB/s | 127 kB 00:00 2024-03-29T11:18:21.127 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:21.128 INFO:teuthology.orchestra.run.smithi019.stdout:Total 625 kB/s | 127 kB 00:00 2024-03-29T11:18:21.128 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:21.134 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:21.134 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:21.192 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:21.193 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:21.291 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:21.345 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:21.446 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:21.449 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:21.450 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:21.450 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repo Size 2024-03-29T11:18:21.450 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:21.450 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:21.450 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:21.450 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 3.1 M 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 13 M 2024-03-29T11:18:21.451 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:21.932 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:21.978 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:18:21.978 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:21.979 INFO:teuthology.orchestra.run.smithi062.stdout:Total 5.9 MB/s | 3.1 MB 00:00 2024-03-29T11:18:21.979 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:21.988 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:21.988 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:22.064 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:22.064 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:22.195 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:22.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs2 2024-03-29T11:18:22.525 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:22.594 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.619 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:22.619 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:22.619 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:22.619 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:22.619 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:22.619 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:23.029 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:23.133 INFO:teuthology.orchestra.run.smithi019.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:23.190 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:23.191 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:23.191 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:23.270 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:23.789 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:23.790 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:23.790 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:23.790 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:23.790 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:23.790 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:23.850 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:24.008 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:24.009 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 31 k 2024-03-29T11:18:24.010 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 140 k 2024-03-29T11:18:24.010 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:24.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install rbd-nbd 2024-03-29T11:18:24.607 INFO:teuthology.orchestra.run.smithi062.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:15:44 AM UTC. 2024-03-29T11:18:24.761 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-29T11:18:24.762 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:24.762 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:24.762 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:24.762 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout:Install 1 Package 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 171 k 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 494 k 2024-03-29T11:18:24.763 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-29T11:18:25.122 INFO:teuthology.orchestra.run.smithi062.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 479 kB/s | 171 kB 00:00 2024-03-29T11:18:25.122 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:25.123 INFO:teuthology.orchestra.run.smithi062.stdout:Total 476 kB/s | 171 kB 00:00 2024-03-29T11:18:25.123 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-29T11:18:25.131 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-29T11:18:25.131 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-29T11:18:25.191 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-29T11:18:25.191 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-29T11:18:25.347 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-29T11:18:25.440 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:25.891 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:26.148 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:26.148 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:26.148 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-29T11:18:26.148 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:26.148 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:26.148 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-29T11:18:26.423 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:26.473 INFO:teuthology.orchestra.run.smithi019.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 13 kB/s | 31 kB 00:02 2024-03-29T11:18:26.473 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:26.474 INFO:teuthology.orchestra.run.smithi019.stdout:Total 13 kB/s | 31 kB 00:02 2024-03-29T11:18:26.474 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:26.478 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:26.479 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:26.518 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:26.518 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:26.663 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:26.782 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.155 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.427 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:27.427 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:27.427 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:27.427 INFO:teuthology.orchestra.run.smithi019.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:27.427 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:27.427 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:27.591 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librados2 2024-03-29T11:18:28.168 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:28.269 INFO:teuthology.orchestra.run.smithi019.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:28.326 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:28.327 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:28.374 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install librbd1 2024-03-29T11:18:28.945 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:29.044 INFO:teuthology.orchestra.run.smithi019.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:29.099 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:29.100 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:29.100 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:29.265 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rados 2024-03-29T11:18:29.841 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:29.943 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:29.998 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:29.999 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:29.999 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:30.057 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rgw 2024-03-29T11:18:30.637 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:30.739 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:30.795 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:30.796 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:30.796 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:30.848 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-cephfs 2024-03-29T11:18:31.430 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:31.531 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:31.586 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:31.587 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:31.588 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:31.640 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install python3-rbd 2024-03-29T11:18:32.223 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:32.324 INFO:teuthology.orchestra.run.smithi019.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:32.379 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:32.380 INFO:teuthology.orchestra.run.smithi019.stdout:Nothing to do. 2024-03-29T11:18:32.380 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:32.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-fuse 2024-03-29T11:18:33.016 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:18:33.172 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:33.173 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:33.173 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:33.173 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:18:33.173 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:33.173 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 85 k 2024-03-29T11:18:33.173 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 231 k 2024-03-29T11:18:33.174 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:33.384 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 405 kB/s | 85 kB 00:00 2024-03-29T11:18:33.385 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:33.385 INFO:teuthology.orchestra.run.smithi019.stdout:Total 402 kB/s | 85 kB 00:00 2024-03-29T11:18:33.385 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:33.392 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:33.393 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:33.450 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:33.450 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:33.593 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:33.694 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.095 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.367 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:34.367 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:34.367 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:34.367 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:34.367 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:34.367 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:34.618 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-mirror 2024-03-29T11:18:35.203 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:35.362 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout: Package Arch Version Repo Size 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:35.363 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:35.364 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:35.364 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:18:35.364 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:35.364 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 3.1 M 2024-03-29T11:18:35.364 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 13 M 2024-03-29T11:18:35.364 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:35.845 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.4 MB/s | 3.1 MB 00:00 2024-03-29T11:18:35.846 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:35.846 INFO:teuthology.orchestra.run.smithi019.stdout:Total 6.4 MB/s | 3.1 MB 00:00 2024-03-29T11:18:35.846 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:35.856 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:35.856 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:35.932 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:35.932 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:36.380 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:36.446 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:36.471 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:36.471 INFO:teuthology.orchestra.run.smithi019.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:36.471 INFO:teuthology.orchestra.run.smithi019.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:18:36.471 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:36.471 INFO:teuthology.orchestra.run.smithi019.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:18:36.471 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:37.563 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:37.563 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:37.564 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:37.564 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:37.824 DEBUG:teuthology.orchestra.run.smithi019:> sudo yum -y install rbd-nbd 2024-03-29T11:18:38.400 INFO:teuthology.orchestra.run.smithi019.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:16:08 AM UTC. 2024-03-29T11:18:38.557 INFO:teuthology.orchestra.run.smithi019.stdout:Dependencies resolved. 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout: Package Architecture Version Repository Size 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:Installing: 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction Summary 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:================================================================================ 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:Install 1 Package 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:38.559 INFO:teuthology.orchestra.run.smithi019.stdout:Total download size: 171 k 2024-03-29T11:18:38.560 INFO:teuthology.orchestra.run.smithi019.stdout:Installed size: 494 k 2024-03-29T11:18:38.560 INFO:teuthology.orchestra.run.smithi019.stdout:Downloading Packages: 2024-03-29T11:18:38.898 INFO:teuthology.orchestra.run.smithi019.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 505 kB/s | 171 kB 00:00 2024-03-29T11:18:38.899 INFO:teuthology.orchestra.run.smithi019.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:38.899 INFO:teuthology.orchestra.run.smithi019.stdout:Total 503 kB/s | 171 kB 00:00 2024-03-29T11:18:38.899 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction check 2024-03-29T11:18:38.908 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction check succeeded. 2024-03-29T11:18:38.908 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction test 2024-03-29T11:18:38.967 INFO:teuthology.orchestra.run.smithi019.stdout:Transaction test succeeded. 2024-03-29T11:18:38.968 INFO:teuthology.orchestra.run.smithi019.stdout:Running transaction 2024-03-29T11:18:39.116 INFO:teuthology.orchestra.run.smithi019.stdout: Preparing : 1/1 2024-03-29T11:18:39.225 INFO:teuthology.orchestra.run.smithi019.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.673 INFO:teuthology.orchestra.run.smithi019.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.999 INFO:teuthology.orchestra.run.smithi019.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:39.999 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:40.000 INFO:teuthology.orchestra.run.smithi019.stdout:Installed: 2024-03-29T11:18:40.000 INFO:teuthology.orchestra.run.smithi019.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:40.000 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:40.000 INFO:teuthology.orchestra.run.smithi019.stdout:Complete! 2024-03-29T11:18:40.274 DEBUG:teuthology.parallel:result is None 2024-03-29T11:18:40.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:40.489 DEBUG:teuthology.orchestra.run.smithi019:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:40.510 INFO:teuthology.orchestra.run.smithi019.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:40.510 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:40.511 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:40.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:18:40.727 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:18:40.751 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:40.751 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:18:40.751 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:18:40.753 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:18:40.753 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:40.753 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:40.781 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:40.781 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:18:40.823 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:18:40.823 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:40.823 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:40.850 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:40.917 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:40.917 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:18:40.960 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:18:41.030 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:18:41.030 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:41.030 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:41.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:41.123 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:41.123 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:18:41.150 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:18:41.218 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:18:41.219 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:41.219 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:41.245 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:41.312 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:41.312 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:18:41.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:18:41.410 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:18:41.506 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:18:41.507 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:41.510 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /var/log/ceph 2024-03-29T11:18:41.536 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:18:41.537 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:41.577 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:18:41.603 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:18:41.604 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:41.644 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:18:41.681 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:18:41.681 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:18:41.681 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622685', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi019.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGmST/JS+fhMJ4n0lfob5aItaHmAVSIgXeMLzvsxBGyZM1xYrIRbGnPndg+PnnS6xedpMCjj7De3Od6LJY6fDJo=', 'smithi062.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAoMD5AnRoAI8E0Aydsy5qWPX4rpSqlIGnpKho++nzwrit0iEO/vBeomdfg2KdU0flCdTWjSvFvqagARkIeHhmc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:18:41.682 DEBUG:teuthology.orchestra.run.smithi019:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:41.711 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:18:41.725 DEBUG:teuthology.orchestra.run.smithi019:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:41.766 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:18:41.792 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:41.793 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:41.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:41.846 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-29 11:18:36.754536979 +0000 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-29 11:14:38.290543922 +0000 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-29 11:14:38.290543922 +0000 2024-03-29T11:18:41.900 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-29 11:14:38.290543922 +0000 2024-03-29T11:18:41.901 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:41.966 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-29T11:18:41.966 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-29T11:18:41.966 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000197073 s, 2.6 MB/s 2024-03-29T11:18:41.967 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:42.023 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-29 11:18:36.755536969 +0000 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-29 11:14:38.647540271 +0000 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-29 11:14:38.647540271 +0000 2024-03-29T11:18:42.078 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-29 11:14:38.647540271 +0000 2024-03-29T11:18:42.079 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:42.142 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-29T11:18:42.143 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-29T11:18:42.143 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000192301 s, 2.7 MB/s 2024-03-29T11:18:42.144 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:42.200 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-29 11:18:36.755536969 +0000 2024-03-29T11:18:42.258 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-29 11:14:38.995536713 +0000 2024-03-29T11:18:42.259 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-29 11:14:38.995536713 +0000 2024-03-29T11:18:42.259 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-29 11:14:38.995536713 +0000 2024-03-29T11:18:42.259 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:42.325 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-29T11:18:42.325 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-29T11:18:42.325 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000221788 s, 2.3 MB/s 2024-03-29T11:18:42.327 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:42.387 DEBUG:teuthology.orchestra.run.smithi019:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout:Access: 2024-03-29 11:18:36.756536959 +0000 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout:Modify: 2024-03-29 11:14:39.330533288 +0000 2024-03-29T11:18:42.445 INFO:teuthology.orchestra.run.smithi019.stdout:Change: 2024-03-29 11:14:39.330533288 +0000 2024-03-29T11:18:42.446 INFO:teuthology.orchestra.run.smithi019.stdout: Birth: 2024-03-29 11:14:39.330533288 +0000 2024-03-29T11:18:42.446 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:42.511 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records in 2024-03-29T11:18:42.511 INFO:teuthology.orchestra.run.smithi019.stderr:1+0 records out 2024-03-29T11:18:42.511 INFO:teuthology.orchestra.run.smithi019.stderr:512 bytes copied, 0.000179844 s, 2.8 MB/s 2024-03-29T11:18:42.512 DEBUG:teuthology.orchestra.run.smithi019:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:42.569 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:42.569 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:42.569 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:18:42.586 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:18:42.586 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-29 11:18:22.897667337 +0000 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-29 11:15:01.191189632 +0000 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-29 11:15:01.191189632 +0000 2024-03-29T11:18:42.641 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-29 11:15:01.191189632 +0000 2024-03-29T11:18:42.642 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:18:42.707 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-29T11:18:42.708 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-29T11:18:42.708 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000275117 s, 1.9 MB/s 2024-03-29T11:18:42.709 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:18:42.765 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-29 11:18:22.898667322 +0000 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-29 11:15:01.575183542 +0000 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-29 11:15:01.575183542 +0000 2024-03-29T11:18:42.820 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-29 11:15:01.575183542 +0000 2024-03-29T11:18:42.821 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:18:42.886 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-29T11:18:42.886 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-29T11:18:42.886 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000165413 s, 3.1 MB/s 2024-03-29T11:18:42.887 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:18:42.942 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-29 11:18:22.898667322 +0000 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-29 11:15:01.964177374 +0000 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-29 11:15:01.964177374 +0000 2024-03-29T11:18:42.998 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-29 11:15:01.964177374 +0000 2024-03-29T11:18:42.999 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:18:43.063 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-29T11:18:43.063 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-29T11:18:43.063 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000171473 s, 3.0 MB/s 2024-03-29T11:18:43.064 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:18:43.119 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-29 11:18:22.898667322 +0000 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-29 11:15:02.337171459 +0000 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-29 11:15:02.337171459 +0000 2024-03-29T11:18:43.174 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-29 11:15:02.337171459 +0000 2024-03-29T11:18:43.175 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:18:43.240 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-29T11:18:43.240 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-29T11:18:43.240 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000220084 s, 2.3 MB/s 2024-03-29T11:18:43.241 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:18:43.296 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:43.296 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:43.296 INFO:tasks.ceph:Generating config... 2024-03-29T11:18:43.297 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:18:43.298 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:18:43.298 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:18:43.298 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:18:43.298 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:18:43.298 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:18:43.298 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:18:43.298 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:18:43.298 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:18:43.298 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:18:43.298 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:18:43.298 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:18:43.298 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:18:43.298 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:18:43.299 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:18:43.299 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:18:43.300 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:18:43.301 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:18:43.301 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:18:43.301 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:18:43.301 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:18:43.301 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:18:43.301 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:18:43.301 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:18:43.352 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:18:43.355 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:18:43.400 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:43.427 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.19:6789]'), ('mon.c', '[v1:172.21.15.19:6790]'), ('mon.b', '[v1:172.21.15.62:6789]')] 2024-03-29T11:18:43.427 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.19:6789],[v1:172.21.15.19:6790],[v1:172.21.15.62:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:18:43.428 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:43.428 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:43.482 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.19:6789]' --addv c '[v1:172.21.15.19:6790]' --addv b '[v1:172.21.15.62:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: generated fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:setting min_mon_release = quincy 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:epoch 0 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:min_mon_release 17 (quincy) 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:election_strategy: 1 2024-03-29T11:18:43.567 INFO:teuthology.orchestra.run.smithi019.stdout:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:18:43.568 INFO:teuthology.orchestra.run.smithi019.stdout:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:18:43.568 INFO:teuthology.orchestra.run.smithi019.stdout:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:18:43.568 INFO:teuthology.orchestra.run.smithi019.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:18:43.569 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:18:43.583 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7173a58a-f9c8-420e-9665-277a04c8f803... 2024-03-29T11:18:43.584 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:43.626 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout:[global] 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: chdir = "" 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: auth supported = cephx 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: filestore xattr use omap = true 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:43.674 INFO:teuthology.orchestra.run.smithi019.stdout: auth debug = true 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on old message = true 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: ms die on bug = true 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default size = 2 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.675 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd allow pg remap = true 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on too few osds = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: osd default data pool replay window = 5 2024-03-29T11:18:43.676 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: mon allow pool delete = true 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: mon cluster log file level = debug 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: debug asserts on shutdown = true 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: mon health detail to clog = false 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: mon host = "[v1:172.21.15.19:6789],[v1:172.21.15.19:6790],[v1:172.21.15.62:6789]" 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: lockdep = True 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: mon client directed command retry = 5 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: mon election default strategy = 1 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: ms bind msgr2 = False 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: ms inject socket failures = 5000 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: ms type = async 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: fsid = 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:18:43.677 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout:[osd] 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd journal size = 100 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd scrub max interval = 600 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd recover clone overlap = true 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug shutdown = true 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug op order = true 2024-03-29T11:18:43.678 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug trim objects = true 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: osd open classes on start = true 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug pg log writeout = true 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: osd map max advance = 10 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: journal zero on create = true 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:43.679 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: bdev debug aio = true 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug misdirected ops = true 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore block size = 96636764160 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression algorithm = zstd 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore fsck on mount = True 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: bluestore zero block detection = True 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluefs = 20 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: debug bluestore = 20 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: debug osd = 20 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: debug rocksdb = 10 2024-03-29T11:18:43.680 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd class default list = * 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd class load list = * 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd client watch timeout = 120 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd debug verify missing on start = True 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:43.681 INFO:teuthology.orchestra.run.smithi019.stdout: osd objectstore = bluestore 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: osd op queue = debug_random 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr] 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: debug mgr = 20 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:43.682 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout:[mon] 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: debug mon = 20 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: debug paxos = 20 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: debug auth = 20 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: mon data avail warn = 5 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: mon osd prime pg temp = true 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.683 INFO:teuthology.orchestra.run.smithi019.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: # 1m isn't quite enough 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:43.684 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: mon scrub interval = 300 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: mon warn on pool no app = False 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: paxos service trim min = 10 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout:[client] 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: rgw cache enabled = true 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable ops log = true 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: rgw enable usage log = true 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: debug monc = 20 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: debug ms = 1 2024-03-29T11:18:43.685 INFO:teuthology.orchestra.run.smithi019.stdout: debug objecter = 20 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi019.stdout: debug rados = 20 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.a] 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.c] 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi019.stdout:[mon.b] 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi062.stdout: chdir = "" 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi062.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:18:43.686 INFO:teuthology.orchestra.run.smithi062.stdout: auth supported = cephx 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: filestore xattr use omap = true 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: mon clock drift allowed = 1.000 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: osd crush chooseleaf type = 0 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: auth debug = true 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.687 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on old message = true 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: ms die on bug = true 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: mon pg warn max object skew = 0 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default size = 2 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow primary affinity = true 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd allow pg remap = true 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:18:43.688 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on no sortbitwise = false 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on too few osds = false 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: mon_allow_pool_size_one = true 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:18:43.689 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: osd default data pool replay window = 5 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: mon allow pool delete = true 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: mon cluster log file level = debug 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: debug asserts on shutdown = true 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: mon health detail to clog = false 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: mon host = "[v1:172.21.15.19:6789],[v1:172.21.15.19:6790],[v1:172.21.15.62:6789]" 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: lockdep = True 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: mon client directed command retry = 5 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: mon election default strategy = 1 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: ms bind msgr2 = False 2024-03-29T11:18:43.690 INFO:teuthology.orchestra.run.smithi062.stdout: ms inject socket failures = 5000 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: ms type = async 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout:[osd] 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: osd journal size = 100 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub load threshold = 5.0 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: osd scrub max interval = 600 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:18:43.691 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd recover clone overlap = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd recovery max chunk = 1048576 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug shutdown = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug op order = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify stray on activate = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug trim objects = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd open classes on start = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug pg log writeout = true 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:18:43.692 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: osd map max advance = 10 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: journal zero on create = true 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: filestore apply finisher threads = 3 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: bdev debug aio = true 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug misdirected ops = true 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore block size = 96636764160 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore compression algorithm = zstd 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore compression mode = aggressive 2024-03-29T11:18:43.693 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore fsck on mount = True 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: bluestore zero block detection = True 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluefs = 20 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: debug bluestore = 20 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: debug monc = 20 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: debug osd = 20 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: debug rocksdb = 10 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd full ratio = 0.9 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: osd class default list = * 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: osd class load list = * 2024-03-29T11:18:43.694 INFO:teuthology.orchestra.run.smithi062.stdout: osd client watch timeout = 120 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify cached snaps = True 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd debug verify missing on start = True 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd mclock override recovery settings = True 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd objectstore = bluestore 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd op queue = debug_random 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd op queue cut off = debug_random 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: osd shutdown pgref assert = True 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr] 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: debug mgr = 20 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-29T11:18:43.695 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mgr/telemetry/nag = false 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout:[mon] 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: debug mon = 20 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: debug paxos = 20 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: debug auth = 20 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon data avail warn = 5 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon mgr mkfs grace = 240 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd reporter subtree level = osd 2024-03-29T11:18:43.696 INFO:teuthology.orchestra.run.smithi062.stdout: mon osd prime pg temp = true 2024-03-29T11:18:43.697 INFO:teuthology.orchestra.run.smithi062.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:18:43.697 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.697 INFO:teuthology.orchestra.run.smithi062.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:18:43.697 INFO:teuthology.orchestra.run.smithi062.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.698 INFO:teuthology.orchestra.run.smithi062.stdout: # 1m isn't quite enough 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon min osdmap epochs = 50 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon osdmap full prune interval = 2 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon osdmap full prune min = 15 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon scrub interval = 300 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: mon warn on pool no app = False 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: paxos service trim min = 10 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout:[client] 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: rgw cache enabled = true 2024-03-29T11:18:43.699 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable ops log = true 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: rgw enable usage log = true 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: debug monc = 20 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: debug ms = 1 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: debug objecter = 20 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout: debug rados = 20 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.a] 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.c] 2024-03-29T11:18:43.700 INFO:teuthology.orchestra.run.smithi062.stdout:[mon.b] 2024-03-29T11:18:43.701 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:18:43.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:18:43.757 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:18:43.757 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:43.758 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:18:43.772 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:43.772 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:18:43.826 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:18:43.827 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:43.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:43.827 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:43.905 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:18:43.905 DEBUG:teuthology.orchestra.run.smithi019:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:43.960 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:18:43.960 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:43.960 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:18:43.960 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:18:44.000 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:18:44.000 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:44.055 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:18:44.056 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:18:44.056 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:44.110 INFO:teuthology.orchestra.run.smithi019.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:18:44.113 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:44.177 INFO:teuthology.orchestra.run.smithi062.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:18:44.179 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:18:44.180 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:18:44.180 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:44.221 INFO:teuthology.orchestra.run.smithi019.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:18:44.237 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:44.278 INFO:teuthology.orchestra.run.smithi062.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:18:44.294 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:18:44.294 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi019.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi062.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:18:44.294 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:44.321 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:44.321 INFO:tasks.ceph:role: osd.0 2024-03-29T11:18:44.321 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:18:44.321 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:44.394 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:44.395 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:44.917 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-29T11:18:44.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-29T11:18:44.919 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:45.048 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:45.084 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:45.153 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:45.153 INFO:tasks.ceph:role: osd.1 2024-03-29T11:18:45.154 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:18:45.154 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:18:45.228 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:45.229 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:45.685 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-29T11:18:45.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-29T11:18:45.687 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:45.778 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:45.808 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:45.875 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:45.875 INFO:tasks.ceph:role: osd.2 2024-03-29T11:18:45.875 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:18:45.875 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:18:45.949 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:45.950 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:46.440 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-29T11:18:46.442 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-29T11:18:46.442 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:46.594 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:46.625 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:46.691 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:46.691 INFO:tasks.ceph:role: osd.3 2024-03-29T11:18:46.691 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:18:46.691 DEBUG:teuthology.orchestra.run.smithi019:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:18:46.765 INFO:teuthology.orchestra.run.smithi019.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:46.766 INFO:teuthology.orchestra.run.smithi019.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:18:47.301 INFO:teuthology.orchestra.run.smithi019.stdout:Discarding blocks...Done. 2024-03-29T11:18:47.303 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi019.front.sepia.ceph.com -o noatime 2024-03-29T11:18:47.303 DEBUG:teuthology.orchestra.run.smithi019:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:47.455 DEBUG:teuthology.orchestra.run.smithi019:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:47.490 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:47.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:47.610+0000 7f940c4c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:18:47.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:47.610+0000 7f940c4c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:18:47.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:47.610+0000 7f940c4c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:47.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:47.610+0000 7f940c4c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:47.587 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:47.610+0000 7f940c4c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:18:47.588 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:47.611+0000 7f940c4c4640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:18:50.436 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:18:50.467 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:50.581+0000 7f2abd60d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:18:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:50.581+0000 7f2abd60d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:18:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:50.581+0000 7f2abd60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:50.581+0000 7f2abd60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:50.558 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:50.581+0000 7f2abd60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:18:50.559 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:50.582+0000 7f2abd60d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:18:53.409 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:18:53.440 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:53.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:53.556+0000 7f2a06299640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:18:53.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:53.556+0000 7f2a06299640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:18:53.533 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:53.556+0000 7f2a06299640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:53.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:53.556+0000 7f2a06299640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:53.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:53.556+0000 7f2a06299640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:18:53.534 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:53.557+0000 7f2a06299640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:18:56.384 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:18:56.414 DEBUG:teuthology.orchestra.run.smithi019:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:18:56.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:56.530+0000 7f328b519640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:18:56.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:56.530+0000 7f328b519640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:18:56.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:56.530+0000 7f328b519640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:56.507 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:56.531+0000 7f328b519640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:56.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:56.531+0000 7f328b519640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:18:56.508 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:18:56.531+0000 7f328b519640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:18:59.364 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:18:59.395 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:18:59.427 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:18:59.428 INFO:tasks.ceph:role: osd.4 2024-03-29T11:18:59.428 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:18:59.428 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:18:59.505 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:18:59.505 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:18:59.505 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:18:59.505 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:18:59.505 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:18:59.506 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:18:59.506 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:18:59.506 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:18:59.506 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:18:59.506 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:00.022 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-29T11:19:00.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-29T11:19:00.024 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:00.156 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:00.194 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:00.263 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:00.263 INFO:tasks.ceph:role: osd.5 2024-03-29T11:19:00.263 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:19:00.263 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:00.338 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:00.338 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:00.339 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:00.802 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-29T11:19:00.804 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-29T11:19:00.804 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:00.901 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:00.935 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:01.005 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:01.005 INFO:tasks.ceph:role: osd.6 2024-03-29T11:19:01.005 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:19:01.005 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:01.081 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:01.081 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:01.082 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:01.564 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-29T11:19:01.566 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-29T11:19:01.566 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:01.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:01.757 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:01.826 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:01.826 INFO:tasks.ceph:role: osd.7 2024-03-29T11:19:01.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:19:01.826 DEBUG:teuthology.orchestra.run.smithi062:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:01.902 INFO:teuthology.orchestra.run.smithi062.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:01.902 INFO:teuthology.orchestra.run.smithi062.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:01.903 INFO:teuthology.orchestra.run.smithi062.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:01.903 INFO:teuthology.orchestra.run.smithi062.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:01.903 INFO:teuthology.orchestra.run.smithi062.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:01.903 INFO:teuthology.orchestra.run.smithi062.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:01.903 INFO:teuthology.orchestra.run.smithi062.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:02.416 INFO:teuthology.orchestra.run.smithi062.stdout:Discarding blocks...Done. 2024-03-29T11:19:02.418 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi062.front.sepia.ceph.com -o noatime 2024-03-29T11:19:02.418 DEBUG:teuthology.orchestra.run.smithi062:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:02.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:02.585 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:02.684 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:02.708+0000 7f254e014640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:19:02.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:02.708+0000 7f254e014640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:19:02.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:02.708+0000 7f254e014640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:02.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:02.708+0000 7f254e014640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:02.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:02.708+0000 7f254e014640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:19:02.685 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:02.709+0000 7f254e014640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:19:05.533 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:19:05.564 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:05.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:05.679+0000 7fc4b6a13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:19:05.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:05.679+0000 7fc4b6a13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:19:05.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:05.679+0000 7fc4b6a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:05.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:05.679+0000 7fc4b6a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:05.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:05.679+0000 7fc4b6a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:19:05.657 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:05.680+0000 7fc4b6a13640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:19:08.508 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:19:08.539 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:08.652+0000 7f50c77bb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:08.652+0000 7f50c77bb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:08.652+0000 7f50c77bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:08.652+0000 7f50c77bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:08.652+0000 7f50c77bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:19:08.631 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:08.653+0000 7f50c77bb640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:19:11.483 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:19:11.514 DEBUG:teuthology.orchestra.run.smithi062:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:11.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:11.628+0000 7fad25ea0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:19:11.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:11.628+0000 7fad25ea0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:19:11.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:11.628+0000 7fad25ea0640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:11.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:11.628+0000 7fad25ea0640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:11.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:11.628+0000 7fad25ea0640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:19:11.606 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:11.629+0000 7fad25ea0640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:19:14.452 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:19:14.483 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:19:14.483 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:14.484 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:19:14.516 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:14.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:19:14.588 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:14.588 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:19:14.658 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:14.658 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:19:14.728 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:14.728 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:19:14.799 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:14.799 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:19:14.831 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:14.831 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:19:14.902 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:14.902 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:19:14.972 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:14.972 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:19:15.041 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:15.041 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:19:15.109 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:15.109 DEBUG:teuthology.orchestra.run.smithi019:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:19:15.127 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:15.127 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:19:15.164 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:19:15.165 DEBUG:teuthology.orchestra.run.smithi019:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:15.170 DEBUG:teuthology.orchestra.run.smithi062:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.y] 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCUowZmDC0ICBAAdxCLoZcUQZJuN7LKvgcIHQ== 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.0] 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCXowZm7UJuJBAAFFz+NCxi/lkyUxfYsGCo1g== 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.1] 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCaowZmCy2yIhAAb3nmXexqGhcXnwEEQzdAqg== 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.2] 2024-03-29T11:19:15.208 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCdowZm7Pk3IRAAjOAN3LYGPt8fLw8EO32n8w== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.3] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCgowZmvY6sHxAANeUS75Rb8OT69a8ZLfh2rA== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[mgr.x] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCUowZmABkEDBAAAGS8egkzbTQPlYApQ5p3jg== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.4] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCmowZmQqk7KhAAjoiZPlihxMySB6IG833dWw== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.5] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCpowZmCoKSKBAA73iwv4JOjCUfFIxgMkDZwA== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.6] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCsowZmOff7JhAAfr6a6UpLBB7bXd/q7YG2sA== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[osd.7] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCvowZmICCNJRAA3t6NrsQA/q+BR5nFIhIZRw== 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout:[client.0] 2024-03-29T11:19:15.209 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCUowZmpk+qDhAAbmgcxdSSZfe0RM7FZuD8PA== 2024-03-29T11:19:15.210 INFO:teuthology.orchestra.run.smithi019.stdout:[client.1] 2024-03-29T11:19:15.210 INFO:teuthology.orchestra.run.smithi019.stdout: key = AQCUowZmH8ANEhAAxsyrIzwGQAq7JNEhoMamig== 2024-03-29T11:19:15.231 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.y] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCUowZmDC0ICBAAdxCLoZcUQZJuN7LKvgcIHQ== 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.0] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCXowZm7UJuJBAAFFz+NCxi/lkyUxfYsGCo1g== 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.1] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCaowZmCy2yIhAAb3nmXexqGhcXnwEEQzdAqg== 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.2] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCdowZm7Pk3IRAAjOAN3LYGPt8fLw8EO32n8w== 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.3] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCgowZmvY6sHxAANeUS75Rb8OT69a8ZLfh2rA== 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout:[mgr.x] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCUowZmABkEDBAAAGS8egkzbTQPlYApQ5p3jg== 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.4] 2024-03-29T11:19:15.232 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCmowZmQqk7KhAAjoiZPlihxMySB6IG833dWw== 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.5] 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCpowZmCoKSKBAA73iwv4JOjCUfFIxgMkDZwA== 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.6] 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCsowZmOff7JhAAfr6a6UpLBB7bXd/q7YG2sA== 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout:[osd.7] 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCvowZmICCNJRAA3t6NrsQA/q+BR5nFIhIZRw== 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCUowZmpk+qDhAAbmgcxdSSZfe0RM7FZuD8PA== 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout:[client.1] 2024-03-29T11:19:15.233 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQCUowZmH8ANEhAAxsyrIzwGQAq7JNEhoMamig== 2024-03-29T11:19:15.234 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:15.254 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:15.326 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.329 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.381 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.383 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.439 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.441 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.491 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.494 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.545 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:15.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:19:15.598 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.601 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.654 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.656 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.707 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.710 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.761 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.763 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:19:15.814 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:15.816 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:15.871 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:15.873 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:19:15.927 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:19:15.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:15.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:16.050 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:19:16.076 DEBUG:teuthology.orchestra.run.smithi019:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:16.143 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:16.240 DEBUG:teuthology.orchestra.run.smithi019:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:19:16.266 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:16.294 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:16.392 DEBUG:teuthology.orchestra.run.smithi062:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:19:16.419 DEBUG:teuthology.orchestra.run.smithi019:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:16.421 DEBUG:teuthology.orchestra.run.smithi062:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:16.473 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:19:16.474 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:19:16.474 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:19:16.476 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:19:16.476 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:19:16.476 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:19:16.478 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:19:16.478 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:19:16.478 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:19:16.515 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:19:16.515 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:19:16.515 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:19:16.516 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:19:16.518 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:19:16.518 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:19:16.518 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:19:16.520 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:19:16.520 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:19:16.520 DEBUG:teuthology.orchestra.run.smithi019:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:19:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.645+0000 7f6d943d6640 10 monclient: get_monmap_and_config 2024-03-29T11:19:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.645+0000 7f6d943d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.645+0000 7f6d943d6640 10 monclient: monmap: 2024-03-29T11:19:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-29T11:19:16.622 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:19:16.646699+0000 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:19:16.646699+0000 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:16.623 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:16.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.646+0000 7f6d943d6640 1 Processor -- start 2024-03-29T11:19:16.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.646+0000 7f6d943d6640 1 -- start start 2024-03-29T11:19:16.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.646+0000 7f6d943d6640 10 monclient: init 2024-03-29T11:19:16.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d8c0b8240 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d8c0b1f60 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c09c8c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c09bd80 con 0x7f6d8c0b8240 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c09b960 con 0x7f6d8c0b1f60 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0b2ec0 con 0x7f6d8c09c8c0 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d943d6640 10 monclient(hunting): authenticate will time out at 876.910381s 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d92973640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f6d8c09c8c0 0x7f6d8c14d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:53442/0 (socket says 172.21.15.19:53442) 2024-03-29T11:19:16.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:16.647+0000 7f6d92973640 1 -- 172.21.15.19:0/3069122053 learned_addr learned my addr 172.21.15.19:0/3069122053 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:19.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): tick 2024-03-29T11:19:19.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:19:19.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 monclient(hunting): continuing hunt 2024-03-29T11:19:19.624 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 -- 172.21.15.19:0/3069122053 >> v1:172.21.15.19:6790/0 conn(0x7f6d8c09c8c0 legacy=0x7f6d8c14d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 -- 172.21.15.19:0/3069122053 >> v1:172.21.15.62:6789/0 conn(0x7f6d8c0b1f60 legacy=0x7f6d8c09c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 -- 172.21.15.19:0/3069122053 >> v1:172.21.15.19:6789/0 conn(0x7f6d8c0b8240 legacy=0x7f6d8c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6d700036e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): picked mon.noname-a con 0x7f6d700072d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c09c8c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 -- 172.21.15.19:0/3069122053 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c13fb60 con 0x7f6d700072d0 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 -- 172.21.15.19:0/3069122053 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d7000f8f0 con 0x7f6d700036e0 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 1 -- 172.21.15.19:0/3069122053 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d70010b30 con 0x7f6d8c09c8c0 2024-03-29T11:19:19.625 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:19.648+0000 7f6d83fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.799 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.821+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573303875 0 0) 0x7f6d8c13fb60 con 0x7f6d700072d0 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.821+0000 7f6d91170640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.821+0000 7f6d91170640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.821+0000 7f6d91170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.821+0000 7f6d91170640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d78003520 con 0x7f6d700072d0 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582466290 0 0) 0x7f6d70010b30 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d8c13fb60 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.800 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2213231602 0 0) 0x7f6d8c13fb60 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 >> v1:172.21.15.62:6789/0 conn(0x7f6d700036e0 legacy=0x7f6d70005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 >> v1:172.21.15.19:6789/0 conn(0x7f6d700072d0 legacy=0x7f6d70009740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 monclient: found mon.noname-b 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d70011d70 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6d84001990 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d943d6640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d943d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.801 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d91170640 10 monclient: dump: 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.822+0000 7f6d943d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:21.802 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d84001af0 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d91170640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d91170640 1 -- 172.21.15.19:0/3069122053 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6d84001f00 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d91170640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d91170640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d91170640 10 monclient: dump: 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:21.803 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:21.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d943d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:21.804 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.823+0000 7f6d943d6640 4 set_mon_vals no callback set 2024-03-29T11:19:21.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 10 monclient: shutdown 2024-03-29T11:19:21.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:21.808 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 1 -- 172.21.15.19:0/3069122053 >> v1:172.21.15.19:6790/0 conn(0x7f6d8c09c8c0 legacy=0x7f6d7000d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 1 -- 172.21.15.19:0/3069122053 shutdown_connections 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 1 -- 172.21.15.19:0/3069122053 >> 172.21.15.19:0/3069122053 conn(0x7f6d8c066b00 msgr2=0x7f6d8c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 1 -- 172.21.15.19:0/3069122053 shutdown_connections 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.824+0000 7f6d943d6640 1 -- 172.21.15.19:0/3069122053 wait complete. 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.825+0000 7f6d943d6640 10 monclient: build_initial_monmap 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.825+0000 7f6d943d6640 10 monclient: monmap: 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:21.809 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.825+0000 7f6d943d6640 1 librados: starting msgr at 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.825+0000 7f6d943d6640 1 librados: starting objecter 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.826+0000 7f6d943d6640 1 Processor -- start 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.826+0000 7f6d943d6640 1 -- start start 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.826+0000 7f6d943d6640 1 librados: setting wanted keys 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.826+0000 7f6d943d6640 1 librados: calling monclient init 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.826+0000 7f6d943d6640 10 monclient: init 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:21.810 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d700036e0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d700072d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d8c09c8c0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0b0b20 con 0x7f6d700036e0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0b17d0 con 0x7f6d8c09c8c0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d8c0b1a40 con 0x7f6d700072d0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d943d6640 10 monclient(hunting): authenticate will time out at 882.090167s 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d91971640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f6d700072d0 0x7f6d8c1db0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:53484/0 (socket says 172.21.15.19:53484) 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d91971640 1 -- 172.21.15.19:0/3534077299 learned_addr learned my addr 172.21.15.19:0/3534077299 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d92973640 1 --1- 172.21.15.19:0/3534077299 >> v1:172.21.15.62:6789/0 conn(0x7f6d8c09c8c0 0x7f6d8c1e1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:40718/0 (socket says 172.21.15.19:40718) 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 client.?.objecter ms_handle_connect 0x7f6d700072d0 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.811 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 client.?.objecter ms_handle_connect 0x7f6d8c09c8c0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 client.?.objecter ms_handle_connect 0x7f6d700036e0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060882682 0 0) 0x7f6d8c0b0b20 con 0x7f6d700036e0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.827+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d70010320 con 0x7f6d700036e0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4019863966 0 0) 0x7f6d70010320 con 0x7f6d700036e0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d8c0b0b20 con 0x7f6d700036e0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6d7c001840 con 0x7f6d700036e0 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.812 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1680088510 0 0) 0x7f6d8c0b0b20 con 0x7f6d700036e0 2024-03-29T11:19:21.813 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 >> v1:172.21.15.19:6790/0 conn(0x7f6d700072d0 legacy=0x7f6d8c1db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 >> v1:172.21.15.62:6789/0 conn(0x7f6d8c09c8c0 legacy=0x7f6d8c1e1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 monclient: found mon.a 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c0b06d0 con 0x7f6d700036e0 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 5 monclient: authenticate success, global_id 4113 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 10 monclient: _renew_subs 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:21.814 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c139af0 con 0x7f6d700036e0 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 10 monclient: _renew_subs 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c1dab50 con 0x7f6d700036e0 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d943d6640 1 librados: init done 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d7c0035f0 con 0x7f6d700036e0 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6d7c003900 con 0x7f6d700036e0 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d9096f640 4 set_mon_vals no callback set 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 monclient: dump: 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:21.815 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6d7c003df0 con 0x7f6d700036e0 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3217103543 0 0) 0x7f6d7c005320 con 0x7f6d700036e0 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 10 client.4113.objecter ms_dispatch 0x7f6d8c000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.828+0000 7f6d82ffd640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T11:19:21.816 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.829+0000 7f6d943d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.829+0000 7f6d943d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.829+0000 7f6d943d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.829+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d74005080 con 0x7f6d700036e0 2024-03-29T11:19:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.833+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6d7c004340 con 0x7f6d700036e0 2024-03-29T11:19:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.833+0000 7f6d82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:21.817 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.833+0000 7f6d82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:21.913 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.937+0000 7f6d943d6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:21.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.937+0000 7f6d943d6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:21.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.937+0000 7f6d943d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:21.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.937+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6d74005330 con 0x7f6d700036e0 2024-03-29T11:19:21.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.938+0000 7f6d82ffd640 1 -- 172.21.15.19:0/3534077299 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6d7c0045c0 con 0x7f6d700036e0 2024-03-29T11:19:21.914 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.938+0000 7f6d82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:19:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.938+0000 7f6d82ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:19:21.915 INFO:teuthology.orchestra.run.smithi019.stderr:adjusted tunables profile to default 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 10 librados: watch_flush enter 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 10 librados: watch_flush exit 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 10 client.4113.objecter successfully canceled tick 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 10 monclient: shutdown 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 >> v1:172.21.15.19:6789/0 conn(0x7f6d700036e0 legacy=0x7f6d8c0acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 shutdown_connections 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 >> 172.21.15.19:0/3534077299 conn(0x7f6d8c066b00 msgr2=0x7f6d8c0b4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:21.916 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 shutdown_connections 2024-03-29T11:19:21.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 1 -- 172.21.15.19:0/3534077299 wait complete. 2024-03-29T11:19:21.917 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:21.939+0000 7f6d943d6640 1 librados: shutdown 2024-03-29T11:19:21.926 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:19:21.927 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:19:21.927 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:21.927 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:19:21.954 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:21.954 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:19:22.021 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:22.021 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:19:22.088 DEBUG:teuthology.orchestra.run.smithi019:> set -ex 2024-03-29T11:19:22.088 DEBUG:teuthology.orchestra.run.smithi019:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:19:22.155 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:22.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:19:22.182 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:22.182 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:19:22.249 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:22.249 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:19:22.313 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:22.337+0000 7fca0e213040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:22.315 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-29T11:19:22.315 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:19:22.316 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:22.338+0000 7fa621645040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:19:22.369 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:22.393+0000 7fca0e213040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:22.372 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:22.394+0000 7fa621645040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:19:22.382 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new e330fe46-e3ef-468b-8a2e-b8c79f658692 0 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.545+0000 7ff596def640 10 monclient: get_monmap_and_config 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.545+0000 7ff596def640 10 monclient: build_initial_monmap 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.545+0000 7ff596def640 10 monclient: monmap: 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:22.546709+0000 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:22.546709+0000 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:22.523 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 1 Processor -- start 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 1 -- start start 2024-03-29T11:19:22.524 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient: init 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient(hunting): picked mon.noname-b con 0x7ff59009ac40 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient(hunting): picked mon.noname-c con 0x7ff59009c870 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient(hunting): picked mon.noname-a con 0x7ff5900b4360 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59009dec0 con 0x7ff5900b4360 2024-03-29T11:19:22.534 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59009da30 con 0x7ff59009c870 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5900b39c0 con 0x7ff59009ac40 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff596def640 10 monclient(hunting): authenticate will time out at 882.825507s 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff58ffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff59009c870 0x7ff59013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43902/0 (socket says 172.21.15.62:43902) 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.546+0000 7ff58ffff640 1 -- 172.21.15.62:0/4107567855 learned_addr learned my addr 172.21.15.62:0/4107567855 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595862962 0 0) 0x7ff59009da30 con 0x7ff59009c870 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): my global_id is 4099 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff580003520 con 0x7ff59009c870 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416950525 0 0) 0x7ff5900b39c0 con 0x7ff59009ac40 2024-03-29T11:19:22.535 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59009da30 con 0x7ff59009ac40 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052341702 0 0) 0x7ff588001490 con 0x7ff5900b4360 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5900b39c0 con 0x7ff5900b4360 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3174394818 0 0) 0x7ff580003520 con 0x7ff59009c870 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 >> v1:172.21.15.19:6790/0 conn(0x7ff59009ac40 legacy=0x7ff5900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.536 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 >> v1:172.21.15.19:6789/0 conn(0x7ff5900b4360 legacy=0x7ff59014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5900b5160 con 0x7ff59009c870 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff57c001690 con 0x7ff59009c870 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: dump: 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.537 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 5 monclient: authenticate success, global_id 4099 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff57c001870 con 0x7ff59009c870 2024-03-29T11:19:22.538 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 1 -- 172.21.15.62:0/4107567855 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff57c001c80 con 0x7ff59009c870 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff58f7fe640 10 monclient: dump: 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.539 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 4 set_mon_vals no callback set 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 10 monclient: shutdown 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.547+0000 7ff596def640 1 -- 172.21.15.62:0/4107567855 >> v1:172.21.15.62:6789/0 conn(0x7ff59009c870 legacy=0x7ff59013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 -- 172.21.15.62:0/4107567855 shutdown_connections 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 -- 172.21.15.62:0/4107567855 >> 172.21.15.62:0/4107567855 conn(0x7ff590066b00 msgr2=0x7ff590066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 -- 172.21.15.62:0/4107567855 shutdown_connections 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 -- 172.21.15.62:0/4107567855 wait complete. 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 10 monclient: build_initial_monmap 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 10 monclient: monmap: 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:22.540 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 librados: starting msgr at 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 librados: starting objecter 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.548+0000 7ff596def640 1 Processor -- start 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 1 -- start start 2024-03-29T11:19:22.541 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 1 librados: setting wanted keys 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 1 librados: calling monclient init 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient: init 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient(hunting): picked mon.noname-b con 0x7ff59009ac40 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient(hunting): picked mon.noname-a con 0x7ff590140660 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient(hunting): picked mon.noname-c con 0x7ff590140f80 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5901423a0 con 0x7ff590140660 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff590140140 con 0x7ff59009ac40 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5900b8040 con 0x7ff590140f80 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff596def640 10 monclient(hunting): authenticate will time out at 882.828599s 2024-03-29T11:19:22.542 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff594b8b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff59009ac40 0x7ff5901d7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43916/0 (socket says 172.21.15.62:43916) 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff594b8b640 1 -- 172.21.15.62:0/3109497617 learned_addr learned my addr 172.21.15.62:0/3109497617 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff58d7fa640 10 client.?.objecter ms_handle_connect 0x7ff59009ac40 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff58d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff58d7fa640 10 client.?.objecter ms_handle_connect 0x7ff590140660 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.549+0000 7ff58d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 client.?.objecter ms_handle_connect 0x7ff590140f80 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965783639 0 0) 0x7ff590140140 con 0x7ff59009ac40 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): my global_id is 4102 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574004eb0 con 0x7ff59009ac40 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59358042 0 0) 0x7ff5901423a0 con 0x7ff590140660 2024-03-29T11:19:22.543 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff590140140 con 0x7ff590140660 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202074458 0 0) 0x7ff5900b8040 con 0x7ff590140f80 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5901423a0 con 0x7ff590140f80 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305167968 0 0) 0x7ff590140140 con 0x7ff590140660 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5900b8040 con 0x7ff590140660 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff57c001b80 con 0x7ff590140660 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.544 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 97473237 0 0) 0x7ff5901423a0 con 0x7ff590140f80 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff590140140 con 0x7ff590140f80 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 858729022 0 0) 0x7ff574004eb0 con 0x7ff59009ac40 2024-03-29T11:19:22.545 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5901423a0 con 0x7ff59009ac40 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff588001b40 con 0x7ff590140f80 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.546 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff584001930 con 0x7ff59009ac40 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.547 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1741220928 0 0) 0x7ff5900b8040 con 0x7ff590140660 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 >> v1:172.21.15.19:6790/0 conn(0x7ff590140f80 legacy=0x7ff5901e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 >> v1:172.21.15.62:6789/0 conn(0x7ff59009ac40 legacy=0x7ff5901d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 monclient: found mon.a 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5900b81d0 con 0x7ff590140660 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:22.548 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 5 monclient: authenticate success, global_id 4119 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 10 monclient: _renew_subs 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5901d7b90 con 0x7ff590140660 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 10 monclient: _renew_subs 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5900b19f0 con 0x7ff590140660 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff596def640 1 librados: init done 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff57c001d60 con 0x7ff590140660 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff57c004d90 con 0x7ff590140660 2024-03-29T11:19:22.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58effd640 4 set_mon_vals no callback set 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 monclient: dump: 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.550 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff57c005280 con 0x7ff590140660 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3731184929 0 0) 0x7ff57c006750 con 0x7ff590140660 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 10 client.4119.objecter ms_dispatch 0x7ff590000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.550+0000 7ff58d7fa640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.551+0000 7ff596def640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.551+0000 7ff596def640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.551+0000 7ff596def640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.551+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff570005080 con 0x7ff590140660 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.553+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff57c006a30 con 0x7ff590140660 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.553+0000 7ff58d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.551 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.553+0000 7ff58d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:22.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.626+0000 7ff596def640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e330fe46-e3ef-468b-8a2e-b8c79f658692", "id": 0}] 2024-03-29T11:19:22.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.626+0000 7ff596def640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e330fe46-e3ef-468b-8a2e-b8c79f658692", "id": 0}] 2024-03-29T11:19:22.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.626+0000 7ff596def640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.603 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.626+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e330fe46-e3ef-468b-8a2e-b8c79f658692", "id": 0} v 0) v1 -- 0x7ff5700058b0 con 0x7ff590140660 2024-03-29T11:19:22.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.628+0000 7ff58d7fa640 1 -- 172.21.15.62:0/3109497617 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e330fe46-e3ef-468b-8a2e-b8c79f658692", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 974446849 0 732707515) 0x7ff57c006bd0 con 0x7ff590140660 2024-03-29T11:19:22.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.628+0000 7ff58d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e330fe46-e3ef-468b-8a2e-b8c79f658692", "id": 0}] 2024-03-29T11:19:22.605 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.628+0000 7ff58d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:22.605 INFO:teuthology.orchestra.run.smithi062.stdout:0 2024-03-29T11:19:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 10 librados: watch_flush enter 2024-03-29T11:19:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 10 librados: watch_flush exit 2024-03-29T11:19:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-29T11:19:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 10 client.4119.objecter successfully canceled tick 2024-03-29T11:19:22.607 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 10 monclient: shutdown 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 >> v1:172.21.15.19:6789/0 conn(0x7ff590140660 legacy=0x7ff5900b1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 shutdown_connections 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 >> 172.21.15.62:0/3109497617 conn(0x7ff590066b00 msgr2=0x7ff59013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 shutdown_connections 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 1 -- 172.21.15.62:0/3109497617 wait complete. 2024-03-29T11:19:22.608 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.629+0000 7ff596def640 1 librados: shutdown 2024-03-29T11:19:22.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 73c12c41-ab79-4175-80b3-80ebd03e7499 1 2024-03-29T11:19:22.720 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.743+0000 7fa71bc63640 10 monclient: get_monmap_and_config 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.743+0000 7fa71bc63640 10 monclient: build_initial_monmap 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.743+0000 7fa71bc63640 10 monclient: monmap: 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:22.744838+0000 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:22.744838+0000 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:22.721 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:22.722 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:22.722 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.744+0000 7fa71bc63640 1 Processor -- start 2024-03-29T11:19:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.744+0000 7fa71bc63640 1 -- start start 2024-03-29T11:19:22.731 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.744+0000 7fa71bc63640 10 monclient: init 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient(hunting): picked mon.noname-a con 0x7fa71409ac40 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient(hunting): picked mon.noname-c con 0x7fa71409c870 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient(hunting): picked mon.noname-b con 0x7fa7140b4360 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71409dec0 con 0x7fa71409ac40 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71409da30 con 0x7fa71409c870 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7140b39c0 con 0x7fa7140b4360 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient(hunting): authenticate will time out at 883.024003s 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7191fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa71409c870 0x7fa71413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43922/0 (socket says 172.21.15.62:43922) 2024-03-29T11:19:22.732 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7191fe640 1 -- 172.21.15.62:0/3462884131 learned_addr learned my addr 172.21.15.62:0/3462884131 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623591868 0 0) 0x7fa71409da30 con 0x7fa71409c870 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa700003520 con 0x7fa71409c870 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661073070 0 0) 0x7fa710001110 con 0x7fa71409ac40 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa71409da30 con 0x7fa71409ac40 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092701978 0 0) 0x7fa7140b39c0 con 0x7fa7140b4360 2024-03-29T11:19:22.733 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa710001110 con 0x7fa7140b4360 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766543022 0 0) 0x7fa700003520 con 0x7fa71409c870 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 >> v1:172.21.15.19:6790/0 conn(0x7fa7140b4360 legacy=0x7fa71414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 >> v1:172.21.15.19:6789/0 conn(0x7fa71409ac40 legacy=0x7fa7140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 monclient: found mon.noname-c 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.734 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140b5160 con 0x7fa71409c870 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: _finish_auth 0 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa704001690 con 0x7fa71409c870 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: dump: 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.735 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa704001870 con 0x7fa71409c870 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 1 -- 172.21.15.62:0/3462884131 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa704001c80 con 0x7fa71409c870 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa7189fd640 10 monclient: dump: 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.736 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.745+0000 7fa71bc63640 4 set_mon_vals no callback set 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 10 monclient: shutdown 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 1 -- 172.21.15.62:0/3462884131 >> v1:172.21.15.62:6789/0 conn(0x7fa71409c870 legacy=0x7fa71413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 1 -- 172.21.15.62:0/3462884131 shutdown_connections 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 1 -- 172.21.15.62:0/3462884131 >> 172.21.15.62:0/3462884131 conn(0x7fa714066b00 msgr2=0x7fa714066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.737 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 1 -- 172.21.15.62:0/3462884131 shutdown_connections 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.746+0000 7fa71bc63640 1 -- 172.21.15.62:0/3462884131 wait complete. 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 10 monclient: build_initial_monmap 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 10 monclient: monmap: 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:22.738 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 1 librados: starting msgr at 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 1 librados: starting objecter 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 1 Processor -- start 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 1 -- start start 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 1 librados: setting wanted keys 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 1 librados: calling monclient init 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.747+0000 7fa71bc63640 10 monclient: init 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient(hunting): picked mon.noname-c con 0x7fa71409ac40 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient(hunting): picked mon.noname-b con 0x7fa7140b6090 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient(hunting): picked mon.noname-a con 0x7fa7140b0f40 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7140b5c00 con 0x7fa7140b0f40 2024-03-29T11:19:22.739 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7140b73b0 con 0x7fa7140b6090 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa71413a7f0 con 0x7fa71409ac40 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa71bc63640 10 monclient(hunting): authenticate will time out at 883.027135s 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa7191fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa7140b6090 0x7fa7141399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43928/0 (socket says 172.21.15.62:43928) 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa7191fe640 1 -- 172.21.15.62:0/4283565710 learned_addr learned my addr 172.21.15.62:0/4283565710 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 client.?.objecter ms_handle_connect 0x7fa7140b6090 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 client.?.objecter ms_handle_connect 0x7fa71409ac40 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 client.?.objecter ms_handle_connect 0x7fa7140b0f40 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 953390175 0 0) 0x7fa7140b73b0 con 0x7fa7140b6090 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.740 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6f8004eb0 con 0x7fa7140b6090 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898200066 0 0) 0x7fa7140b5c00 con 0x7fa7140b0f40 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7140b73b0 con 0x7fa7140b0f40 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4149505394 0 0) 0x7fa71413a7f0 con 0x7fa71409ac40 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.741 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7140b5c00 con 0x7fa71409ac40 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 594389862 0 0) 0x7fa6f8004eb0 con 0x7fa7140b6090 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa71413a7f0 con 0x7fa7140b6090 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa704001bc0 con 0x7fa7140b6090 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.742 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 819521502 0 0) 0x7fa7140b73b0 con 0x7fa7140b0f40 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6f8004eb0 con 0x7fa7140b0f40 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa70c001b50 con 0x7fa7140b0f40 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.743 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324507870 0 0) 0x7fa7140b5c00 con 0x7fa71409ac40 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7140b73b0 con 0x7fa71409ac40 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1928688959 0 0) 0x7fa71413a7f0 con 0x7fa7140b6090 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.748+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 >> v1:172.21.15.19:6790/0 conn(0x7fa71409ac40 legacy=0x7fa7140b7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 >> v1:172.21.15.19:6789/0 conn(0x7fa7140b0f40 legacy=0x7fa71413a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 monclient: found mon.b 2024-03-29T11:19:22.744 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7140b7620 con 0x7fa7140b6090 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 monclient: _renew_subs 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7140b5840 con 0x7fa7140b6090 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 monclient: _renew_subs 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.745 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7141428e0 con 0x7fa7140b6090 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 1 librados: init done 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa704004bf0 con 0x7fa7140b6090 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa704004f00 con 0x7fa7140b6090 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 monclient: dump: 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.746 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70b7fe640 4 set_mon_vals no callback set 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa7040053f0 con 0x7fa7140b6090 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2693986524 0 0) 0x7fa7040067f0 con 0x7fa7140b6090 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 10 client.4108.objecter ms_dispatch 0x7fa714000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 3 client.4108.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 3 client.4108.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa70a7fc640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.749+0000 7fa71bc63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.747 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.750+0000 7fa71bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.750+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6fc005080 con 0x7fa7140b6090 2024-03-29T11:19:22.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.752+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa704006a00 con 0x7fa7140b6090 2024-03-29T11:19:22.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.752+0000 7fa70a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.748 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.752+0000 7fa70a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:22.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.835+0000 7fa71bc63640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "73c12c41-ab79-4175-80b3-80ebd03e7499", "id": 1}] 2024-03-29T11:19:22.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.835+0000 7fa71bc63640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "73c12c41-ab79-4175-80b3-80ebd03e7499", "id": 1}] 2024-03-29T11:19:22.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.835+0000 7fa71bc63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.812 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.835+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd new", "uuid": "73c12c41-ab79-4175-80b3-80ebd03e7499", "id": 1} v 0) v1 -- 0x7fa6fc0058b0 con 0x7fa7140b6090 2024-03-29T11:19:22.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.839+0000 7fa70a7fc640 1 -- 172.21.15.62:0/4283565710 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "73c12c41-ab79-4175-80b3-80ebd03e7499", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1399362763 0 940483276) 0x7fa7040057c0 con 0x7fa7140b6090 2024-03-29T11:19:22.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.839+0000 7fa70a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "73c12c41-ab79-4175-80b3-80ebd03e7499", "id": 1}] 2024-03-29T11:19:22.816 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.839+0000 7fa70a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:22.817 INFO:teuthology.orchestra.run.smithi062.stdout:1 2024-03-29T11:19:22.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 10 librados: watch_flush enter 2024-03-29T11:19:22.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 10 librados: watch_flush exit 2024-03-29T11:19:22.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T11:19:22.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 10 client.4108.objecter successfully canceled tick 2024-03-29T11:19:22.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 10 monclient: shutdown 2024-03-29T11:19:22.818 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 >> v1:172.21.15.62:6789/0 conn(0x7fa7140b6090 legacy=0x7fa7141399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 shutdown_connections 2024-03-29T11:19:22.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 >> 172.21.15.62:0/4283565710 conn(0x7fa714066b00 msgr2=0x7fa71413c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 shutdown_connections 2024-03-29T11:19:22.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 1 -- 172.21.15.62:0/4283565710 wait complete. 2024-03-29T11:19:22.819 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.840+0000 7fa71bc63640 1 librados: shutdown 2024-03-29T11:19:22.827 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new fc3f3fea-ba5e-4e6a-9313-446861b7a8a1 2 2024-03-29T11:19:22.924 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:22.948+0000 7fca0e213040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.950+0000 7fb872b2b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.950+0000 7fb872b2b640 10 monclient: build_initial_monmap 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.950+0000 7fb872b2b640 10 monclient: monmap: 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:22.951753+0000 2024-03-29T11:19:22.927 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:22.951753+0000 2024-03-29T11:19:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:22.928 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:22.928 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 1 Processor -- start 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 1 -- start start 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient: init 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb86c09ac40 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb86c09c870 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb86c0b4360 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c09dec0 con 0x7fb86c09c870 2024-03-29T11:19:22.936 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c09da30 con 0x7fb86c09ac40 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0b39c0 con 0x7fb86c0b4360 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb872b2b640 10 monclient(hunting): authenticate will time out at 883.230588s 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb8708c7640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb86c09ac40 0x7fb86c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43942/0 (socket says 172.21.15.62:43942) 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.951+0000 7fb8708c7640 1 -- 172.21.15.62:0/3061871638 learned_addr learned my addr 172.21.15.62:0/3061871638 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300587228 0 0) 0x7fb86c09da30 con 0x7fb86c09ac40 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c003520 con 0x7fb86c09ac40 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466236918 0 0) 0x7fb858001110 con 0x7fb86c09c870 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.937 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c09da30 con 0x7fb86c09c870 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163770091 0 0) 0x7fb86c0b39c0 con 0x7fb86c0b4360 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858001110 con 0x7fb86c0b4360 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1841154451 0 0) 0x7fb85c003520 con 0x7fb86c09ac40 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 >> v1:172.21.15.19:6790/0 conn(0x7fb86c0b4360 legacy=0x7fb86c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 >> v1:172.21.15.19:6789/0 conn(0x7fb86c09c870 legacy=0x7fb86c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.938 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0b5160 con 0x7fb86c09ac40 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb860001690 con 0x7fb86c09ac40 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: dump: 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.939 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb872b2b640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb872b2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb860001870 con 0x7fb86c09ac40 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 1 -- 172.21.15.62:0/3061871638 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb860001c80 con 0x7fb86c09ac40 2024-03-29T11:19:22.940 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb872b2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb872b2b640 4 set_mon_vals no callback set 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.952+0000 7fb86b7fe640 10 monclient: dump: 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.941 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 10 monclient: shutdown 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 -- 172.21.15.62:0/3061871638 >> v1:172.21.15.62:6789/0 conn(0x7fb86c09ac40 legacy=0x7fb86c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 -- 172.21.15.62:0/3061871638 shutdown_connections 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 -- 172.21.15.62:0/3061871638 >> 172.21.15.62:0/3061871638 conn(0x7fb86c066b00 msgr2=0x7fb86c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 -- 172.21.15.62:0/3061871638 shutdown_connections 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 -- 172.21.15.62:0/3061871638 wait complete. 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 10 monclient: build_initial_monmap 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 10 monclient: monmap: 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:22.942 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 librados: starting msgr at 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.953+0000 7fb872b2b640 1 librados: starting objecter 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 Processor -- start 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 -- start start 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 librados: setting wanted keys 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 librados: calling monclient init 2024-03-29T11:19:22.943 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient: init 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient(hunting): picked mon.noname-a con 0x7fb86c09ac40 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient(hunting): picked mon.noname-c con 0x7fb86c1ccb70 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient(hunting): picked mon.noname-b con 0x7fb86c1425f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c1cd520 con 0x7fb86c09ac40 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0acd00 con 0x7fb86c1425f0 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c1cd000 con 0x7fb86c1ccb70 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb872b2b640 10 monclient(hunting): authenticate will time out at 883.233672s 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb8710c8640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb86c1425f0 0x7fb86c1e29d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43956/0 (socket says 172.21.15.62:43956) 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb8708c7640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fb86c09ac40 0x7fb86c1da690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.62:59266/0 (socket says 172.21.15.62:59266) 2024-03-29T11:19:22.944 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.954+0000 7fb8710c8640 1 -- 172.21.15.62:0/513604214 learned_addr learned my addr 172.21.15.62:0/513604214 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.?.objecter ms_handle_connect 0x7fb86c1425f0 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.?.objecter ms_handle_connect 0x7fb86c09ac40 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.?.objecter ms_handle_connect 0x7fb86c1ccb70 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704945884 0 0) 0x7fb86c0acd00 con 0x7fb86c1425f0 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb850004eb0 con 0x7fb86c1425f0 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195025836 0 0) 0x7fb86c1cd520 con 0x7fb86c09ac40 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.945 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c0acd00 con 0x7fb86c09ac40 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313533542 0 0) 0x7fb86c1cd000 con 0x7fb86c1ccb70 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c1cd520 con 0x7fb86c1ccb70 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 288937025 0 0) 0x7fb86c0acd00 con 0x7fb86c09ac40 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c1cd000 con 0x7fb86c09ac40 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069856678 0 0) 0x7fb850004eb0 con 0x7fb86c1425f0 2024-03-29T11:19:22.946 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c0acd00 con 0x7fb86c1425f0 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb860001b80 con 0x7fb86c09ac40 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): dump: 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.947 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb864001c70 con 0x7fb86c1425f0 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): dump: 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.948 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 852940137 0 0) 0x7fb86c1cd520 con 0x7fb86c1ccb70 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb850004eb0 con 0x7fb86c1ccb70 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb8580018a0 con 0x7fb86c1ccb70 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient(hunting): dump: 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.949 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2615680014 0 0) 0x7fb86c1cd000 con 0x7fb86c09ac40 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 >> v1:172.21.15.19:6790/0 conn(0x7fb86c1ccb70 legacy=0x7fb86c13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 >> v1:172.21.15.62:6789/0 conn(0x7fb86c1425f0 legacy=0x7fb86c1e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 monclient: found mon.a 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0b1af0 con 0x7fb86c09ac40 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 5 monclient: authenticate success, global_id 4131 2024-03-29T11:19:22.950 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 10 monclient: _renew_subs 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c1cc300 con 0x7fb86c09ac40 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 10 monclient: _renew_subs 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c1daf80 con 0x7fb86c09ac40 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb872b2b640 1 librados: init done 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb860001d40 con 0x7fb86c09ac40 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb8600010e0 con 0x7fb86c09ac40 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 monclient: dump: 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:22.951 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb86affd640 4 set_mon_vals no callback set 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb860005120 con 0x7fb86c09ac40 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2073049444 0 0) 0x7fb8600067c0 con 0x7fb86c09ac40 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 10 client.4131.objecter ms_dispatch 0x7fb86c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:19:22.952 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.955+0000 7fb8697fa640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.956+0000 7fb872b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.956+0000 7fb872b2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.956+0000 7fb872b2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.956+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c005080 con 0x7fb86c09ac40 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.958+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb860006450 con 0x7fb86c09ac40 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.958+0000 7fb8697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:22.953 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:22.958+0000 7fb8697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:22.953 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:22.974+0000 7fa621645040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:19:22.981 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.004+0000 7fca0e213040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:23.008 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.031+0000 7fa621645040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:19:23.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.032+0000 7fb872b2b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fc3f3fea-ba5e-4e6a-9313-446861b7a8a1", "id": 2}] 2024-03-29T11:19:23.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.032+0000 7fb872b2b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fc3f3fea-ba5e-4e6a-9313-446861b7a8a1", "id": 2}] 2024-03-29T11:19:23.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.032+0000 7fb872b2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:23.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.032+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd new", "uuid": "fc3f3fea-ba5e-4e6a-9313-446861b7a8a1", "id": 2} v 0) v1 -- 0x7fb84c0058b0 con 0x7fb86c09ac40 2024-03-29T11:19:23.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.033+0000 7fb8697fa640 1 -- 172.21.15.62:0/513604214 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc3f3fea-ba5e-4e6a-9313-446861b7a8a1", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4093201570 0 216599125) 0x7fb860006a00 con 0x7fb86c09ac40 2024-03-29T11:19:23.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.033+0000 7fb8697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fc3f3fea-ba5e-4e6a-9313-446861b7a8a1", "id": 2}] 2024-03-29T11:19:23.010 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.033+0000 7fb8697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:23.011 INFO:teuthology.orchestra.run.smithi062.stdout:2 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 10 librados: watch_flush enter 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 10 librados: watch_flush exit 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 10 client.4131.objecter successfully canceled tick 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 10 monclient: shutdown 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 >> v1:172.21.15.19:6789/0 conn(0x7fb86c09ac40 legacy=0x7fb86c1da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 shutdown_connections 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 >> 172.21.15.62:0/513604214 conn(0x7fb86c066b00 msgr2=0x7fb86c066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 shutdown_connections 2024-03-29T11:19:23.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 1 -- 172.21.15.62:0/513604214 wait complete. 2024-03-29T11:19:23.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.034+0000 7fb872b2b640 1 librados: shutdown 2024-03-29T11:19:23.022 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5 3 2024-03-29T11:19:23.027 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.051+0000 7fca0e213040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:23.056 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.079+0000 7fa621645040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:19:23.083 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.106+0000 7fca0e213040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:23.114 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.137+0000 7fa621645040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.145+0000 7fb6517ed640 10 monclient: get_monmap_and_config 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.145+0000 7fb6517ed640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.145+0000 7fb6517ed640 10 monclient: monmap: 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:23.146542+0000 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:23.146542+0000 2024-03-29T11:19:23.122 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.123 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.123 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.123 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:23.123 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:23.123 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.130 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.150+0000 7fca0e213040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:23.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.145+0000 7fb6517ed640 1 Processor -- start 2024-03-29T11:19:23.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.145+0000 7fb6517ed640 1 -- start start 2024-03-29T11:19:23.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.145+0000 7fb6517ed640 10 monclient: init 2024-03-29T11:19:23.132 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient(hunting): picked mon.noname-a con 0x7fb64c09ac40 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient(hunting): picked mon.noname-b con 0x7fb64c09c870 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient(hunting): picked mon.noname-c con 0x7fb64c0b4360 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c09dec0 con 0x7fb64c09ac40 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c09da30 con 0x7fb64c0b4360 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c0b39c0 con 0x7fb64c09c870 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb6517ed640 10 monclient(hunting): authenticate will time out at 883.425312s 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb64b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb64c0b4360 0x7fb64c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43972/0 (socket says 172.21.15.62:43972) 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb64b7fe640 1 -- 172.21.15.62:0/1956402736 learned_addr learned my addr 172.21.15.62:0/1956402736 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242244229 0 0) 0x7fb64c09da30 con 0x7fb64c0b4360 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.133 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb634003520 con 0x7fb64c0b4360 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638952898 0 0) 0x7fb64c0b39c0 con 0x7fb64c09c870 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c09da30 con 0x7fb64c09c870 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951411502 0 0) 0x7fb640001110 con 0x7fb64c09ac40 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.134 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.146+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c0b39c0 con 0x7fb64c09ac40 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065360347 0 0) 0x7fb634003520 con 0x7fb64c0b4360 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 >> v1:172.21.15.19:6790/0 conn(0x7fb64c09c870 legacy=0x7fb64c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 >> v1:172.21.15.19:6789/0 conn(0x7fb64c09ac40 legacy=0x7fb64c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c0b5160 con 0x7fb64c0b4360 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.135 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb644001a10 con 0x7fb64c0b4360 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: dump: 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.136 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb644001bf0 con 0x7fb64c0b4360 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 1 -- 172.21.15.62:0/1956402736 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb644002040 con 0x7fb64c0b4360 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 4 set_mon_vals no callback set 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb649ffb640 10 monclient: dump: 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.137 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 10 monclient: shutdown 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 1 -- 172.21.15.62:0/1956402736 >> v1:172.21.15.62:6789/0 conn(0x7fb64c0b4360 legacy=0x7fb64c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 1 -- 172.21.15.62:0/1956402736 shutdown_connections 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 1 -- 172.21.15.62:0/1956402736 >> 172.21.15.62:0/1956402736 conn(0x7fb64c066b00 msgr2=0x7fb64c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 1 -- 172.21.15.62:0/1956402736 shutdown_connections 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.147+0000 7fb6517ed640 1 -- 172.21.15.62:0/1956402736 wait complete. 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.148+0000 7fb6517ed640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.138 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.148+0000 7fb6517ed640 10 monclient: monmap: 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.148+0000 7fb6517ed640 1 librados: starting msgr at 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.148+0000 7fb6517ed640 1 librados: starting objecter 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.148+0000 7fb6517ed640 1 Processor -- start 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 1 -- start start 2024-03-29T11:19:23.139 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 1 librados: setting wanted keys 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 1 librados: calling monclient init 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient: init 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient(hunting): picked mon.noname-b con 0x7fb64c09ac40 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient(hunting): picked mon.noname-c con 0x7fb64c143800 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient(hunting): picked mon.noname-a con 0x7fb64c1db250 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c145ca0 con 0x7fb64c1db250 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c1da6b0 con 0x7fb64c09ac40 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb64c144bb0 con 0x7fb64c143800 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6517ed640 10 monclient(hunting): authenticate will time out at 883.428460s 2024-03-29T11:19:23.140 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb64affd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb64c09ac40 0x7fb64c1444a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43982/0 (socket says 172.21.15.62:43982) 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb64affd640 1 -- 172.21.15.62:0/214855788 learned_addr learned my addr 172.21.15.62:0/214855788 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 client.?.objecter ms_handle_connect 0x7fb64c09ac40 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 client.?.objecter ms_handle_connect 0x7fb64c1db250 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 client.?.objecter ms_handle_connect 0x7fb64c143800 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248643118 0 0) 0x7fb64c1da6b0 con 0x7fb64c09ac40 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb638004eb0 con 0x7fb64c09ac40 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275238367 0 0) 0x7fb64c145ca0 con 0x7fb64c1db250 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.141 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.149+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c1da6b0 con 0x7fb64c1db250 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828409073 0 0) 0x7fb64c144bb0 con 0x7fb64c143800 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c145ca0 con 0x7fb64c143800 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1579219504 0 0) 0x7fb638004eb0 con 0x7fb64c09ac40 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb64c144bb0 con 0x7fb64c09ac40 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408186796 0 0) 0x7fb64c1da6b0 con 0x7fb64c1db250 2024-03-29T11:19:23.142 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb638004eb0 con 0x7fb64c1db250 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb640001980 con 0x7fb64c09ac40 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): dump: 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:23.143 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663129929 0 0) 0x7fb64c145ca0 con 0x7fb64c143800 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb64c1da6b0 con 0x7fb64c143800 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb644001f00 con 0x7fb64c1db250 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): dump: 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.144 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb63c0018b0 con 0x7fb64c143800 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient(hunting): dump: 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.145 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3624130489 0 0) 0x7fb64c144bb0 con 0x7fb64c09ac40 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 >> v1:172.21.15.19:6790/0 conn(0x7fb64c143800 legacy=0x7fb64c1dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 >> v1:172.21.15.19:6789/0 conn(0x7fb64c1db250 legacy=0x7fb64c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 monclient: found mon.b 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64c140ca0 con 0x7fb64c09ac40 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:19:23.146 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 10 monclient: _renew_subs 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64c146360 con 0x7fb64c09ac40 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 10 monclient: _renew_subs 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64c1e35c0 con 0x7fb64c09ac40 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6517ed640 1 librados: init done 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb640001ae0 con 0x7fb64c09ac40 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb6400038a0 con 0x7fb64c09ac40 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6497fa640 4 set_mon_vals no callback set 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 monclient: dump: 2024-03-29T11:19:23.147 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb640003e50 con 0x7fb64c09ac40 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2249373769 0 0) 0x7fb6400066f0 con 0x7fb64c09ac40 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 10 client.4120.objecter ms_dispatch 0x7fb64c000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:19:23.148 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 3 client.4120.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 3 client.4120.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.150+0000 7fb6337fe640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.151+0000 7fb6517ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.151+0000 7fb6517ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.151+0000 7fb6517ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.151+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c005080 con 0x7fb64c09ac40 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.153+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb640004340 con 0x7fb64c09ac40 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.153+0000 7fb6337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.149 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.153+0000 7fb6337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:23.160 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.183+0000 7fa621645040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:19:23.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.226+0000 7fb6517ed640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5", "id": 3}] 2024-03-29T11:19:23.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.226+0000 7fb6517ed640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5", "id": 3}] 2024-03-29T11:19:23.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.226+0000 7fb6517ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.205 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.226+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5", "id": 3} v 0) v1 -- 0x7fb62c0058b0 con 0x7fb64c09ac40 2024-03-29T11:19:23.207 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.230+0000 7fb6337fe640 1 -- 172.21.15.62:0/214855788 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3108458957 0 525046306) 0x7fb640006330 con 0x7fb64c09ac40 2024-03-29T11:19:23.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.230+0000 7fb6337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5", "id": 3}] 2024-03-29T11:19:23.208 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.230+0000 7fb6337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:23.208 INFO:teuthology.orchestra.run.smithi062.stdout:3 2024-03-29T11:19:23.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 10 librados: watch_flush enter 2024-03-29T11:19:23.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 10 librados: watch_flush exit 2024-03-29T11:19:23.209 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 10 monclient: shutdown 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 >> v1:172.21.15.62:6789/0 conn(0x7fb64c09ac40 legacy=0x7fb64c1444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 shutdown_connections 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 >> 172.21.15.62:0/214855788 conn(0x7fb64c066b00 msgr2=0x7fb64c13e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 shutdown_connections 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 1 -- 172.21.15.62:0/214855788 wait complete. 2024-03-29T11:19:23.210 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.231+0000 7fb6517ed640 1 librados: shutdown 2024-03-29T11:19:23.219 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 70842ebb-d0e8-4235-9ac9-dad2b393a0f0 4 2024-03-29T11:19:23.247 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.271+0000 7fca0e213040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:23.281 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.304+0000 7fa621645040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:19:23.320 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.343+0000 7fca0e213040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:23.356 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.379+0000 7fa621645040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.383+0000 7f9fcd1a8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.383+0000 7f9fcd1a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.383+0000 7f9fcd1a8640 10 monclient: monmap: 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:23.384574+0000 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:23.384574+0000 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.360 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:23.361 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:23.361 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.368 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.391+0000 7fca0e213040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.383+0000 7f9fcd1a8640 1 Processor -- start 2024-03-29T11:19:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.383+0000 7f9fcd1a8640 1 -- start start 2024-03-29T11:19:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.383+0000 7f9fcd1a8640 10 monclient: init 2024-03-29T11:19:23.369 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fc809ac40 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fc809c870 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fc80b4360 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc809dec0 con 0x7f9fc809ac40 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc809da30 con 0x7f9fc80b4360 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc80b39c0 con 0x7f9fc809c870 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fcd1a8640 10 monclient(hunting): authenticate will time out at 883.663296s 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc759e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9fc80b4360 0x7f9fc814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43994/0 (socket says 172.21.15.62:43994) 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc759e640 1 -- 172.21.15.62:0/1074554088 learned_addr learned my addr 172.21.15.62:0/1074554088 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2621912839 0 0) 0x7f9fc809da30 con 0x7f9fc80b4360 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.370 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb0003520 con 0x7f9fc80b4360 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865083040 0 0) 0x7f9fc80b39c0 con 0x7f9fc809c870 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc809da30 con 0x7f9fc809c870 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159385996 0 0) 0x7f9fb8001110 con 0x7f9fc809ac40 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.384+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc80b39c0 con 0x7f9fc809ac40 2024-03-29T11:19:23.371 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4074565119 0 0) 0x7f9fb0003520 con 0x7f9fc80b4360 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 >> v1:172.21.15.19:6790/0 conn(0x7f9fc809c870 legacy=0x7f9fc813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 >> v1:172.21.15.19:6789/0 conn(0x7f9fc809ac40 legacy=0x7f9fc80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc80b5160 con 0x7f9fc80b4360 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9fc0001a10 con 0x7f9fc80b4360 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.372 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: dump: 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fc0001bf0 con 0x7f9fc80b4360 2024-03-29T11:19:23.373 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 1 -- 172.21.15.62:0/1074554088 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9fc0002040 con 0x7f9fc80b4360 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fc5d9b640 10 monclient: dump: 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.374 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 4 set_mon_vals no callback set 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 10 monclient: shutdown 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.385+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/1074554088 >> v1:172.21.15.62:6789/0 conn(0x7f9fc80b4360 legacy=0x7f9fc814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.386+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/1074554088 shutdown_connections 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.386+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/1074554088 >> 172.21.15.62:0/1074554088 conn(0x7f9fc8066b00 msgr2=0x7f9fc8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.386+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/1074554088 shutdown_connections 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.386+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/1074554088 wait complete. 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.386+0000 7f9fcd1a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.386+0000 7f9fcd1a8640 10 monclient: monmap: 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:23.375 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 librados: starting msgr at 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 librados: starting objecter 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 Processor -- start 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 -- start start 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 librados: setting wanted keys 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 librados: calling monclient init 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient: init 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.376 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fc809ac40 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fc81d70c0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fc81df7f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc81419c0 con 0x7f9fc81df7f0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc813ffa0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc8140db0 con 0x7f9fc809ac40 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.387+0000 7f9fcd1a8640 10 monclient(hunting): authenticate will time out at 883.666773s 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fc659c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f9fc81d70c0 0x7f9fc81d7540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:43996/0 (socket says 172.21.15.62:43996) 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fc659c640 1 -- 172.21.15.62:0/2316667113 learned_addr learned my addr 172.21.15.62:0/2316667113 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 client.?.objecter ms_handle_connect 0x7f9fc81d70c0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 client.?.objecter ms_handle_connect 0x7f9fc81df7f0 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.377 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 client.?.objecter ms_handle_connect 0x7f9fc809ac40 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154527940 0 0) 0x7f9fc813ffa0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fb4004eb0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923608894 0 0) 0x7f9fc81419c0 con 0x7f9fc81df7f0 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc813ffa0 con 0x7f9fc81df7f0 2024-03-29T11:19:23.378 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233330039 0 0) 0x7f9fc8140db0 con 0x7f9fc809ac40 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc81419c0 con 0x7f9fc809ac40 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3447330558 0 0) 0x7f9fb4004eb0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fc8140db0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9fbc0019b0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.379 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4115931808 0 0) 0x7f9fc813ffa0 con 0x7f9fc81df7f0 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fb4004eb0 con 0x7f9fc81df7f0 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3541779583 0 0) 0x7f9fc81419c0 con 0x7f9fc809ac40 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fc813ffa0 con 0x7f9fc809ac40 2024-03-29T11:19:23.380 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9fc0001f00 con 0x7f9fc81df7f0 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9fb8001860 con 0x7f9fc809ac40 2024-03-29T11:19:23.381 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2043775533 0 0) 0x7f9fc8140db0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.382 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 >> v1:172.21.15.19:6790/0 conn(0x7f9fc809ac40 legacy=0x7f9fc81412b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 >> v1:172.21.15.19:6789/0 conn(0x7f9fc81df7f0 legacy=0x7f9fc81e1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 monclient: found mon.b 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc81d7c50 con 0x7f9fc81d70c0 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 10 monclient: _renew_subs 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc81e3300 con 0x7f9fc81d70c0 2024-03-29T11:19:23.383 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 10 monclient: _renew_subs 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc81e3670 con 0x7f9fc81d70c0 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9fcd1a8640 1 librados: init done 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fbc003610 con 0x7f9fc81d70c0 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.388+0000 7f9faf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9fbc003920 con 0x7f9fc81d70c0 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9fc559a640 4 set_mon_vals no callback set 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 10 monclient: dump: 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.384 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9fbc003ed0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1926592467 0 0) 0x7f9fbc0067c0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 10 client.4126.objecter ms_dispatch 0x7f9fc8000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 3 client.4126.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 3 client.4126.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9faf7fe640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:19:23.385 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9fcd1a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9fcd1a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9fcd1a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.389+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa8005080 con 0x7f9fc81d70c0 2024-03-29T11:19:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.391+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9fbc004310 con 0x7f9fc81d70c0 2024-03-29T11:19:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.391+0000 7f9faf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.386 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.391+0000 7f9faf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:23.406 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.429+0000 7fa621645040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:19:23.424 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.445+0000 7fca0e213040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:23.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.465+0000 7f9fcd1a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "70842ebb-d0e8-4235-9ac9-dad2b393a0f0", "id": 4}] 2024-03-29T11:19:23.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.465+0000 7f9fcd1a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "70842ebb-d0e8-4235-9ac9-dad2b393a0f0", "id": 4}] 2024-03-29T11:19:23.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.465+0000 7f9fcd1a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.442 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.465+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd new", "uuid": "70842ebb-d0e8-4235-9ac9-dad2b393a0f0", "id": 4} v 0) v1 -- 0x7f9fa80058b0 con 0x7f9fc81d70c0 2024-03-29T11:19:23.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9faf7fe640 1 -- 172.21.15.62:0/2316667113 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "70842ebb-d0e8-4235-9ac9-dad2b393a0f0", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1736434136 0 1697012583) 0x7f9fbc006260 con 0x7f9fc81d70c0 2024-03-29T11:19:23.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9faf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "70842ebb-d0e8-4235-9ac9-dad2b393a0f0", "id": 4}] 2024-03-29T11:19:23.446 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9faf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:23.446 INFO:teuthology.orchestra.run.smithi062.stdout:4 2024-03-29T11:19:23.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9fcd1a8640 10 librados: watch_flush enter 2024-03-29T11:19:23.447 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9fcd1a8640 10 librados: watch_flush exit 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9fcd1a8640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.469+0000 7f9fcd1a8640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 10 monclient: shutdown 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 >> v1:172.21.15.62:6789/0 conn(0x7f9fc81d70c0 legacy=0x7f9fc81d7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 shutdown_connections 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 >> 172.21.15.62:0/2316667113 conn(0x7f9fc8066b00 msgr2=0x7f9fc813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 shutdown_connections 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 1 -- 172.21.15.62:0/2316667113 wait complete. 2024-03-29T11:19:23.448 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.470+0000 7f9fcd1a8640 1 librados: shutdown 2024-03-29T11:19:23.457 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new 57b4c2d1-f924-44f9-843c-a9eb27b02126 5 2024-03-29T11:19:23.461 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.484+0000 7fa621645040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:19:23.471 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.494+0000 7fca0e213040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:23.512 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.535+0000 7fa621645040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:19:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.584+0000 7ff3de21e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.584+0000 7ff3de21e640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.584+0000 7ff3de21e640 10 monclient: monmap: 2024-03-29T11:19:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.561 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:23.585779+0000 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:23.585779+0000 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:23.562 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 1 Processor -- start 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 1 -- start start 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient: init 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3d80b0ad0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.566 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3d8142320 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3d806f430 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8142780 con 0x7ff3d8142320 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8072690 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d8071230 con 0x7ff3d806f430 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3de21e640 10 monclient(hunting): authenticate will time out at 883.864695s 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3d77fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff3d80b0ad0 0x7ff3d8071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44002/0 (socket says 172.21.15.62:44002) 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.585+0000 7ff3d77fe640 1 -- 172.21.15.62:0/2721253223 learned_addr learned my addr 172.21.15.62:0/2721253223 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263034729 0 0) 0x7ff3d8072690 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.567 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c0003520 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371919547 0 0) 0x7ff3d8142780 con 0x7ff3d8142320 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8072690 con 0x7ff3d8142320 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3843345387 0 0) 0x7ff3d8071230 con 0x7ff3d806f430 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8142780 con 0x7ff3d806f430 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098555334 0 0) 0x7ff3c0003520 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.568 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 >> v1:172.21.15.19:6790/0 conn(0x7ff3d806f430 legacy=0x7ff3d814a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 >> v1:172.21.15.19:6789/0 conn(0x7ff3d8142320 legacy=0x7ff3d806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 monclient: found mon.noname-c 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80714a0 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3c8001690 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3de21e640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3de21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:23.569 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: dump: 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3de21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3c8001870 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.570 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 1 -- 172.21.15.62:0/2721253223 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3c8001c80 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3d67fc640 10 monclient: dump: 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3de21e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:23.571 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.586+0000 7ff3de21e640 4 set_mon_vals no callback set 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 10 monclient: shutdown 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 1 -- 172.21.15.62:0/2721253223 >> v1:172.21.15.62:6789/0 conn(0x7ff3d80b0ad0 legacy=0x7ff3d8071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 1 -- 172.21.15.62:0/2721253223 shutdown_connections 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 1 -- 172.21.15.62:0/2721253223 >> 172.21.15.62:0/2721253223 conn(0x7ff3d80b4970 msgr2=0x7ff3d80b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 1 -- 172.21.15.62:0/2721253223 shutdown_connections 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.587+0000 7ff3de21e640 1 -- 172.21.15.62:0/2721253223 wait complete. 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.588+0000 7ff3de21e640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.588+0000 7ff3de21e640 10 monclient: monmap: 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:23.572 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.588+0000 7ff3de21e640 1 librados: starting msgr at 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.588+0000 7ff3de21e640 1 librados: starting objecter 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.588+0000 7ff3de21e640 1 Processor -- start 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 1 -- start start 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 1 librados: setting wanted keys 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 1 librados: calling monclient init 2024-03-29T11:19:23.573 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 10 monclient: init 2024-03-29T11:19:23.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:23.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 10 monclient(hunting): picked mon.noname-b con 0x7ff3d806f430 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.589+0000 7ff3de21e640 10 monclient(hunting): picked mon.noname-c con 0x7ff3d80b0ad0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient(hunting): picked mon.noname-a con 0x7ff3d8142320 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.578 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d81388a0 con 0x7ff3d8142320 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d80aae40 con 0x7ff3d806f430 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d80ac000 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient(hunting): authenticate will time out at 883.868893s 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3d77fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff3d806f430 0x7ff3d80aa210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44008/0 (socket says 172.21.15.62:44008) 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3d77fe640 1 -- 172.21.15.62:0/2254751211 learned_addr learned my addr 172.21.15.62:0/2254751211 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 client.?.objecter ms_handle_connect 0x7ff3d806f430 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 client.?.objecter ms_handle_connect 0x7ff3d80b0ad0 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285689569 0 0) 0x7ff3d80aae40 con 0x7ff3d806f430 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.579 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3c4004eb0 con 0x7ff3d806f430 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 client.?.objecter ms_handle_connect 0x7ff3d8142320 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058763271 0 0) 0x7ff3d80ac000 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d80aae40 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1496666758 0 0) 0x7ff3d8142780 con 0x7ff3d8142320 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.580 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d80ac000 con 0x7ff3d8142320 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1329889853 0 0) 0x7ff3d80aae40 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d8142780 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3cc0017f0 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): dump: 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.581 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972554733 0 0) 0x7ff3c4004eb0 con 0x7ff3d806f430 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3d80aae40 con 0x7ff3d806f430 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3c8001b80 con 0x7ff3d806f430 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): dump: 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.582 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1580356065 0 0) 0x7ff3d80ac000 con 0x7ff3d8142320 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3c4004eb0 con 0x7ff3d8142320 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3d0001ad0 con 0x7ff3d8142320 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient(hunting): dump: 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.583 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3755788697 0 0) 0x7ff3d8142780 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 >> v1:172.21.15.62:6789/0 conn(0x7ff3d806f430 legacy=0x7ff3d80aa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 >> v1:172.21.15.19:6789/0 conn(0x7ff3d8142320 legacy=0x7ff3d81d6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 monclient: found mon.c 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d80a9440 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.584 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 5 monclient: authenticate success, global_id 4142 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient: _renew_subs 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d80aa920 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient: _renew_subs 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d80adee0 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3de21e640 1 librados: init done 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff3cc001c80 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.585 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.590+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3cc003790 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 10 monclient: dump: 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.586 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3d5ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff3cc003c80 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2738552251 0 0) 0x7ff3cc005440 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 10 client.4142.objecter ms_dispatch 0x7ff3d8000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 3 client.4142.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 3 client.4142.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3bffff640 20 client.4142.objecter dump_active .. 0 homeless 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3de21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3de21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3de21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.591+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b8005080 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.594+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff3cc004fd0 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.594+0000 7ff3bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.587 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.594+0000 7ff3bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:23.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.668+0000 7ff3de21e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "57b4c2d1-f924-44f9-843c-a9eb27b02126", "id": 5}] 2024-03-29T11:19:23.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.668+0000 7ff3de21e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "57b4c2d1-f924-44f9-843c-a9eb27b02126", "id": 5}] 2024-03-29T11:19:23.645 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.668+0000 7ff3de21e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:19:23.646 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.668+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd new", "uuid": "57b4c2d1-f924-44f9-843c-a9eb27b02126", "id": 5} v 0) v1 -- 0x7ff3b80058b0 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.648 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.671+0000 7ff3bffff640 1 -- 172.21.15.62:0/2254751211 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "57b4c2d1-f924-44f9-843c-a9eb27b02126", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3677634689 0 1988412176) 0x7ff3cc005240 con 0x7ff3d80b0ad0 2024-03-29T11:19:23.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.671+0000 7ff3bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "57b4c2d1-f924-44f9-843c-a9eb27b02126", "id": 5}] 2024-03-29T11:19:23.649 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.671+0000 7ff3bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:23.649 INFO:teuthology.orchestra.run.smithi062.stdout:5 2024-03-29T11:19:23.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 10 librados: watch_flush enter 2024-03-29T11:19:23.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 10 librados: watch_flush exit 2024-03-29T11:19:23.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 20 client.4142.objecter shutdown clearing up homeless session... 2024-03-29T11:19:23.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 10 client.4142.objecter successfully canceled tick 2024-03-29T11:19:23.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 10 monclient: shutdown 2024-03-29T11:19:23.650 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 >> v1:172.21.15.19:6790/0 conn(0x7ff3d80b0ad0 legacy=0x7ff3d80ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 shutdown_connections 2024-03-29T11:19:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 >> 172.21.15.62:0/2254751211 conn(0x7ff3d80b4970 msgr2=0x7ff3d806ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 shutdown_connections 2024-03-29T11:19:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.672+0000 7ff3de21e640 1 -- 172.21.15.62:0/2254751211 wait complete. 2024-03-29T11:19:23.651 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.673+0000 7ff3de21e640 1 librados: shutdown 2024-03-29T11:19:23.660 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734 6 2024-03-29T11:19:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.783+0000 7fb80ea33640 10 monclient: get_monmap_and_config 2024-03-29T11:19:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.783+0000 7fb80ea33640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.783+0000 7fb80ea33640 10 monclient: monmap: 2024-03-29T11:19:23.760 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:23.784543+0000 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:23.784543+0000 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:23.761 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.769 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.783+0000 7fb80ea33640 1 Processor -- start 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.783+0000 7fb80ea33640 1 -- start start 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.783+0000 7fb80ea33640 10 monclient: init 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient(hunting): picked mon.noname-a con 0x7fb80809ac40 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient(hunting): picked mon.noname-b con 0x7fb80809c870 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient(hunting): picked mon.noname-c con 0x7fb8080b4360 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80809dec0 con 0x7fb80809ac40 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80809da30 con 0x7fb8080b4360 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080b39c0 con 0x7fb80809c870 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.770 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80ea33640 10 monclient(hunting): authenticate will time out at 884.063361s 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80cfd0640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb8080b4360 0x7fb80814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44012/0 (socket says 172.21.15.62:44012) 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb80cfd0640 1 -- 172.21.15.62:0/3687065975 learned_addr learned my addr 172.21.15.62:0/3687065975 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620527761 0 0) 0x7fb80809da30 con 0x7fb8080b4360 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb806ffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.784+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7f4003520 con 0x7fb8080b4360 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582069569 0 0) 0x7fb8080b39c0 con 0x7fb80809c870 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80809da30 con 0x7fb80809c870 2024-03-29T11:19:23.771 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2233329341 0 0) 0x7fb7f8001110 con 0x7fb80809ac40 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8080b39c0 con 0x7fb80809ac40 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2283484936 0 0) 0x7fb7f4003520 con 0x7fb8080b4360 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 >> v1:172.21.15.19:6790/0 conn(0x7fb80809c870 legacy=0x7fb80813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 >> v1:172.21.15.19:6789/0 conn(0x7fb80809ac40 legacy=0x7fb8080b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8080b5160 con 0x7fb8080b4360 2024-03-29T11:19:23.772 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb800001a10 con 0x7fb8080b4360 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: dump: 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.773 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb800001b70 con 0x7fb8080b4360 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 1 -- 172.21.15.62:0/3687065975 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb800001f80 con 0x7fb8080b4360 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb806ffd640 10 monclient: dump: 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.774 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 4 set_mon_vals no callback set 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 10 monclient: shutdown 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 1 -- 172.21.15.62:0/3687065975 >> v1:172.21.15.62:6789/0 conn(0x7fb8080b4360 legacy=0x7fb80814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 1 -- 172.21.15.62:0/3687065975 shutdown_connections 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 1 -- 172.21.15.62:0/3687065975 >> 172.21.15.62:0/3687065975 conn(0x7fb808066b00 msgr2=0x7fb808066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.775 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 1 -- 172.21.15.62:0/3687065975 shutdown_connections 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.785+0000 7fb80ea33640 1 -- 172.21.15.62:0/3687065975 wait complete. 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.786+0000 7fb80ea33640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.786+0000 7fb80ea33640 10 monclient: monmap: 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:23.776 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.786+0000 7fb80ea33640 1 librados: starting msgr at 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.786+0000 7fb80ea33640 1 librados: starting objecter 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.786+0000 7fb80ea33640 1 Processor -- start 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 1 -- start start 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 1 librados: setting wanted keys 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 1 librados: calling monclient init 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient: init 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient(hunting): picked mon.noname-b con 0x7fb80809ac40 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient(hunting): picked mon.noname-a con 0x7fb808140730 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient(hunting): picked mon.noname-c con 0x7fb8081d8300 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.777 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb808142380 con 0x7fb808140730 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8081d76d0 con 0x7fb80809ac40 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8080b1a60 con 0x7fb8081d8300 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb80ea33640 10 monclient(hunting): authenticate will time out at 884.066540s 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb807fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fb80809ac40 0x7fb808142b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44028/0 (socket says 172.21.15.62:44028) 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb807fff640 1 -- 172.21.15.62:0/4234404056 learned_addr learned my addr 172.21.15.62:0/4234404056 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 client.?.objecter ms_handle_connect 0x7fb80809ac40 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 client.?.objecter ms_handle_connect 0x7fb808140730 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 client.?.objecter ms_handle_connect 0x7fb8081d8300 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643239892 0 0) 0x7fb8081d76d0 con 0x7fb80809ac40 2024-03-29T11:19:23.778 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.787+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e8004eb0 con 0x7fb80809ac40 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102121648 0 0) 0x7fb808142380 con 0x7fb808140730 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8081d76d0 con 0x7fb808140730 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478023505 0 0) 0x7fb8080b1a60 con 0x7fb8081d8300 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.779 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808142380 con 0x7fb8081d8300 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3979657728 0 0) 0x7fb7e8004eb0 con 0x7fb80809ac40 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8080b1a60 con 0x7fb80809ac40 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f8001860 con 0x7fb80809ac40 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.780 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 108921239 0 0) 0x7fb808142380 con 0x7fb8081d8300 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7e8004eb0 con 0x7fb8081d8300 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb800001e80 con 0x7fb8081d8300 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.781 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2481615131 0 0) 0x7fb8081d76d0 con 0x7fb808140730 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb808142380 con 0x7fb808140730 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f00018f0 con 0x7fb808140730 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:23.782 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1290130679 0 0) 0x7fb8080b1a60 con 0x7fb80809ac40 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 >> v1:172.21.15.19:6790/0 conn(0x7fb8081d8300 legacy=0x7fb8081e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 >> v1:172.21.15.19:6789/0 conn(0x7fb808140730 legacy=0x7fb8081d7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.783 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 monclient: found mon.b 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80813ffa0 con 0x7fb80809ac40 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 10 monclient: _renew_subs 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb808146350 con 0x7fb80809ac40 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 10 monclient: _renew_subs 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.784 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb808140d90 con 0x7fb80809ac40 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb80ea33640 1 librados: init done 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7f8003610 con 0x7fb80809ac40 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f8003a20 con 0x7fb80809ac40 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb8067fc640 4 set_mon_vals no callback set 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 monclient: dump: 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.785 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb7f8003fd0 con 0x7fb80809ac40 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2791848578 0 0) 0x7fb7f8006840 con 0x7fb80809ac40 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 10 client.4138.objecter ms_dispatch 0x7fb808000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 3 client.4138.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.788+0000 7fb804ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.789+0000 7fb80ea33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.789+0000 7fb80ea33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.789+0000 7fb80ea33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.786 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.789+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec005080 con 0x7fb80809ac40 2024-03-29T11:19:23.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.791+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb7f8004460 con 0x7fb80809ac40 2024-03-29T11:19:23.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.791+0000 7fb804ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.787 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.791+0000 7fb804ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:23.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.866+0000 7fb80ea33640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734", "id": 6}] 2024-03-29T11:19:23.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.866+0000 7fb80ea33640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734", "id": 6}] 2024-03-29T11:19:23.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.866+0000 7fb80ea33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.843 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.866+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734", "id": 6} v 0) v1 -- 0x7fb7ec0058b0 con 0x7fb80809ac40 2024-03-29T11:19:23.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.869+0000 7fb804ff9640 1 -- 172.21.15.62:0/4234404056 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2577512319 0 1113811849) 0x7fb7f8006a50 con 0x7fb80809ac40 2024-03-29T11:19:23.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.869+0000 7fb804ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734", "id": 6}] 2024-03-29T11:19:23.846 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.869+0000 7fb804ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:23.847 INFO:teuthology.orchestra.run.smithi062.stdout:6 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 10 librados: watch_flush enter 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 10 librados: watch_flush exit 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 10 monclient: shutdown 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 >> v1:172.21.15.62:6789/0 conn(0x7fb80809ac40 legacy=0x7fb808142b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.848 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 shutdown_connections 2024-03-29T11:19:23.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 >> 172.21.15.62:0/4234404056 conn(0x7fb808066b00 msgr2=0x7fb8080b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 shutdown_connections 2024-03-29T11:19:23.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 1 -- 172.21.15.62:0/4234404056 wait complete. 2024-03-29T11:19:23.849 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.870+0000 7fb80ea33640 1 librados: shutdown 2024-03-29T11:19:23.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo ceph --cluster ceph osd new ecb106b2-c5e5-4ff8-ac8e-49090ad27960 7 2024-03-29T11:19:23.868 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:23.892+0000 7fca0e213040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:23.920 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:23.943+0000 7fa621645040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:19:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.981+0000 7fca500a8640 10 monclient: get_monmap_and_config 2024-03-29T11:19:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.981+0000 7fca500a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.958 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.981+0000 7fca500a8640 10 monclient: monmap: 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:19:23.982850+0000 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:19:23.982850+0000 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:23.959 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 1 Processor -- start 2024-03-29T11:19:23.969 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 1 -- start start 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient: init 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient(hunting): picked mon.noname-c con 0x7fca4809ac40 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient(hunting): picked mon.noname-b con 0x7fca4809c870 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient(hunting): picked mon.noname-a con 0x7fca480b4360 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4809dec0 con 0x7fca480b4360 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4809da30 con 0x7fca4809ac40 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480b39c0 con 0x7fca4809c870 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.982+0000 7fca500a8640 10 monclient(hunting): authenticate will time out at 884.261769s 2024-03-29T11:19:23.970 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4de44640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fca4809ac40 0x7fca480b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44036/0 (socket says 172.21.15.62:44036) 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4de44640 1 -- 172.21.15.62:0/2246939873 learned_addr learned my addr 172.21.15.62:0/2246939873 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741542666 0 0) 0x7fca4809da30 con 0x7fca4809ac40 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca34003520 con 0x7fca4809ac40 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530570537 0 0) 0x7fca40001490 con 0x7fca480b4360 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:19:23.971 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca4809da30 con 0x7fca480b4360 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791620736 0 0) 0x7fca480b39c0 con 0x7fca4809c870 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca40001490 con 0x7fca4809c870 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2031827981 0 0) 0x7fca34003520 con 0x7fca4809ac40 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 >> v1:172.21.15.19:6790/0 conn(0x7fca4809c870 legacy=0x7fca4813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 >> v1:172.21.15.19:6789/0 conn(0x7fca480b4360 legacy=0x7fca4814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 monclient: found mon.noname-c 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.972 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca480b5160 con 0x7fca4809ac40 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca500a8640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fca44001690 con 0x7fca4809ac40 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca500a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: dump: 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.973 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca500a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca44001870 con 0x7fca4809ac40 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca500a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 1 -- 172.21.15.62:0/2246939873 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fca44001c80 con 0x7fca4809ac40 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca500a8640 4 set_mon_vals no callback set 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.974 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.983+0000 7fca4ce42640 10 monclient: dump: 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 10 monclient: shutdown 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 1 -- 172.21.15.62:0/2246939873 >> v1:172.21.15.62:6789/0 conn(0x7fca4809ac40 legacy=0x7fca480b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.975 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 1 -- 172.21.15.62:0/2246939873 shutdown_connections 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 1 -- 172.21.15.62:0/2246939873 >> 172.21.15.62:0/2246939873 conn(0x7fca48066b00 msgr2=0x7fca48066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 1 -- 172.21.15.62:0/2246939873 shutdown_connections 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 1 -- 172.21.15.62:0/2246939873 wait complete. 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 10 monclient: build_initial_monmap 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.984+0000 7fca500a8640 10 monclient: monmap: 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 0 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 0.000000 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:created 0.000000 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:23.976 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 librados: starting msgr at 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 librados: starting objecter 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 Processor -- start 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 -- start start 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 librados: setting wanted keys 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 librados: calling monclient init 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient: init 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient(hunting): picked mon.noname-c con 0x7fca4809ac40 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient(hunting): picked mon.noname-b con 0x7fca481c88e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient(hunting): picked mon.noname-a con 0x7fca481cacb0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:23.977 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca481ca2b0 con 0x7fca481cacb0 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca481ca520 con 0x7fca481c88e0 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca481c85c0 con 0x7fca4809ac40 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca500a8640 10 monclient(hunting): authenticate will time out at 884.264680s 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca4d643640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fca481c88e0 0x7fca480b1000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.62:44042/0 (socket says 172.21.15.62:44042) 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.985+0000 7fca4d643640 1 -- 172.21.15.62:0/2389199104 learned_addr learned my addr 172.21.15.62:0/2389199104 (peer_addr_for_me v1:172.21.15.62:0/0) 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.?.objecter ms_handle_connect 0x7fca481c88e0 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.?.objecter ms_handle_connect 0x7fca4809ac40 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.?.objecter ms_handle_connect 0x7fca481cacb0 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:23.978 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493613421 0 0) 0x7fca481ca520 con 0x7fca481c88e0 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca2c004eb0 con 0x7fca481c88e0 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388508540 0 0) 0x7fca481c85c0 con 0x7fca4809ac40 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca481ca520 con 0x7fca4809ac40 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1058867667 0 0) 0x7fca481ca2b0 con 0x7fca481cacb0 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:23.979 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): my global_id is 4161 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca481c85c0 con 0x7fca481cacb0 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 483474153 0 0) 0x7fca2c004eb0 con 0x7fca481c88e0 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca481ca2b0 con 0x7fca481c88e0 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fca38001a10 con 0x7fca481c88e0 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.980 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1615232537 0 0) 0x7fca481ca520 con 0x7fca4809ac40 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca2c004eb0 con 0x7fca4809ac40 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010898840 0 0) 0x7fca481c85c0 con 0x7fca481cacb0 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca481ca520 con 0x7fca481cacb0 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fca44001b80 con 0x7fca4809ac40 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:23.981 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fca40001a40 con 0x7fca481cacb0 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:23.982 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2321043577 0 0) 0x7fca481ca2b0 con 0x7fca481c88e0 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 >> v1:172.21.15.19:6790/0 conn(0x7fca4809ac40 legacy=0x7fca481d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 >> v1:172.21.15.19:6789/0 conn(0x7fca481cacb0 legacy=0x7fca481e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 monclient: found mon.b 2024-03-29T11:19:23.983 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca480b1710 con 0x7fca481c88e0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 10 monclient: _renew_subs 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca480b1960 con 0x7fca481c88e0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 10 monclient: _renew_subs 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca38001c70 con 0x7fca481c88e0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.984 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca48142d50 con 0x7fca481c88e0 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca500a8640 1 librados: init done 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fca38003a60 con 0x7fca481c88e0 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 monclient: dump: 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:epoch 1 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:election_strategy: 1 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:23.985 INFO:teuthology.orchestra.run.smithi062.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3ffff640 4 set_mon_vals no callback set 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fca38003f50 con 0x7fca481c88e0 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3065030769 0 0) 0x7fca38004200 con 0x7fca481c88e0 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 10 client.4144.objecter ms_dispatch 0x7fca48000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 3 client.4144.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.986+0000 7fca3e7fc640 3 client.4144.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.987+0000 7fca3e7fc640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.987+0000 7fca500a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.987+0000 7fca500a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.987+0000 7fca500a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.987+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca4809b020 con 0x7fca481c88e0 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.989+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fca380057c0 con 0x7fca481c88e0 2024-03-29T11:19:23.986 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.989+0000 7fca3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:23.987 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:23.989+0000 7fca3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:24.009 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.033+0000 7fca0e213040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:24.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.063+0000 7fca500a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ecb106b2-c5e5-4ff8-ac8e-49090ad27960", "id": 7}] 2024-03-29T11:19:24.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.063+0000 7fca500a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ecb106b2-c5e5-4ff8-ac8e-49090ad27960", "id": 7}] 2024-03-29T11:19:24.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.063+0000 7fca500a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:24.040 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.063+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ecb106b2-c5e5-4ff8-ac8e-49090ad27960", "id": 7} v 0) v1 -- 0x7fca4813dfe0 con 0x7fca481c88e0 2024-03-29T11:19:24.043 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.066+0000 7fca3e7fc640 1 -- 172.21.15.62:0/2389199104 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ecb106b2-c5e5-4ff8-ac8e-49090ad27960", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2577542519 0 1371665406) 0x7fca380059e0 con 0x7fca481c88e0 2024-03-29T11:19:24.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ecb106b2-c5e5-4ff8-ac8e-49090ad27960", "id": 7}] 2024-03-29T11:19:24.044 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:24.044 INFO:teuthology.orchestra.run.smithi062.stdout:7 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 10 librados: watch_flush enter 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 10 librados: watch_flush exit 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 10 monclient: shutdown 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 >> v1:172.21.15.62:6789/0 conn(0x7fca481c88e0 legacy=0x7fca480b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.045 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 shutdown_connections 2024-03-29T11:19:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 >> 172.21.15.62:0/2389199104 conn(0x7fca48066b00 msgr2=0x7fca480b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.067+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 shutdown_connections 2024-03-29T11:19:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.068+0000 7fca500a8640 1 -- 172.21.15.62:0/2389199104 wait complete. 2024-03-29T11:19:24.046 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-29T11:19:24.068+0000 7fca500a8640 1 librados: shutdown 2024-03-29T11:19:24.055 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:19:24.055 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:19:24.057 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:19:24.057 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:19:24.057 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:19:24.059 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:19:24.059 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:19:24.060 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:19:24.061 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:19:24.062 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:19:24.062 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:19:24.064 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:19:24.064 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:19:24.064 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:19:24.065 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.087+0000 7fa621645040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:19:24.066 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:19:24.066 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:19:24.067 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:19:24.068 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:19:24.068 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:19:24.069 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:19:24.071 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:19:24.071 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:19:24.071 DEBUG:teuthology.orchestra.run.smithi062:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:19:24.073 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:19:24.073 DEBUG:teuthology.orchestra.run.smithi019:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:24.148 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.172+0000 7fca0e213040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.223+0000 7fc7f59c4640 10 monclient: get_monmap_and_config 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.223+0000 7fc7f59c4640 10 monclient: build_initial_monmap 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.223+0000 7fc7f59c4640 10 monclient: monmap: 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:19:24.224606+0000 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:19:24.224606+0000 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:24.200 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:24.201 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:24.201 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:24.201 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:24.201 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:24.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.224+0000 7fc7f59c4640 1 Processor -- start 2024-03-29T11:19:24.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.224+0000 7fc7f59c4640 1 -- start start 2024-03-29T11:19:24.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.224+0000 7fc7f59c4640 10 monclient: init 2024-03-29T11:19:24.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:24.209 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f00b8240 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f00b1f60 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f009c8c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f009bd80 con 0x7fc7f00b1f60 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7ee7fc640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc7f00b1f60 0x7fc7f009c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:60990/0 (socket says 172.21.15.19:60990) 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7ee7fc640 1 -- 172.21.15.19:0/2309416930 learned_addr learned my addr 172.21.15.19:0/2309416930 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f009b960 con 0x7fc7f00b8240 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f00b2ec0 con 0x7fc7f009c8c0 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7f59c4640 10 monclient(hunting): authenticate will time out at 884.488525s 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888836265 0 0) 0x7fc7f009bd80 con 0x7fc7f00b1f60 2024-03-29T11:19:24.210 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d8003520 con 0x7fc7f00b1f60 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758793006 0 0) 0x7fc7f00b2ec0 con 0x7fc7f009c8c0 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.225+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f009bd80 con 0x7fc7f009c8c0 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2672357880 0 0) 0x7fc7d8003520 con 0x7fc7f00b1f60 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 >> v1:172.21.15.19:6790/0 conn(0x7fc7f009c8c0 legacy=0x7fc7f014d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 >> v1:172.21.15.62:6789/0 conn(0x7fc7f00b8240 legacy=0x7fc7f013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:24.211 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 monclient: found mon.noname-a 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f013fb60 con 0x7fc7f00b1f60 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc7e4001690 con 0x7fc7f00b1f60 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: dump: 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:24.212 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7e4001870 con 0x7fc7f00b1f60 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:24.213 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 1 -- 172.21.15.19:0/2309416930 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc7e4001c80 con 0x7fc7f00b1f60 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 4 set_mon_vals no callback set 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7edffb640 10 monclient: dump: 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:24.214 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 10 monclient: shutdown 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 >> v1:172.21.15.19:6789/0 conn(0x7fc7f00b1f60 legacy=0x7fc7f009c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 shutdown_connections 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 >> 172.21.15.19:0/2309416930 conn(0x7fc7f0066b00 msgr2=0x7fc7f0066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 shutdown_connections 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.226+0000 7fc7f59c4640 1 -- 172.21.15.19:0/2309416930 wait complete. 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 10 monclient: build_initial_monmap 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 10 monclient: monmap: 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 0 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:created 0.000000 2024-03-29T11:19:24.215 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 1 librados: starting msgr at 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 1 librados: starting objecter 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 1 Processor -- start 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 1 -- start start 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 1 librados: setting wanted keys 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 1 librados: calling monclient init 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.227+0000 7fc7f59c4640 10 monclient: init 2024-03-29T11:19:24.216 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient(hunting): picked mon.noname-c con 0x7fc7f009c8c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient(hunting): picked mon.noname-b con 0x7fc7f00b8240 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient(hunting): picked mon.noname-a con 0x7fc7f0144a70 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f0136ea0 con 0x7fc7f0144a70 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f0138bc0 con 0x7fc7f00b8240 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f0137ad0 con 0x7fc7f009c8c0 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7f59c4640 10 monclient(hunting): authenticate will time out at 884.491474s 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7ef7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc7f0144a70 0x7fc7f01e2110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:32770/0 (socket says 172.21.15.19:32770) 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7ef7fe640 1 -- 172.21.15.19:0/1057554401 learned_addr learned my addr 172.21.15.19:0/1057554401 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 client.?.objecter ms_handle_connect 0x7fc7f0144a70 2024-03-29T11:19:24.217 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 client.?.objecter ms_handle_connect 0x7fc7f009c8c0 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 client.?.objecter ms_handle_connect 0x7fc7f00b8240 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322522646 0 0) 0x7fc7f0136ea0 con 0x7fc7f0144a70 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7dc004eb0 con 0x7fc7f0144a70 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655959293 0 0) 0x7fc7f0138bc0 con 0x7fc7f00b8240 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.218 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): my global_id is 4171 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0136ea0 con 0x7fc7f00b8240 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170788406 0 0) 0x7fc7f0137ad0 con 0x7fc7f009c8c0 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.228+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0138bc0 con 0x7fc7f009c8c0 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2271555487 0 0) 0x7fc7dc004eb0 con 0x7fc7f0144a70 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7f0137ad0 con 0x7fc7f0144a70 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc7e8001e50 con 0x7fc7f0144a70 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.219 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient(hunting): dump: 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:24.220 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3165794643 0 0) 0x7fc7f0136ea0 con 0x7fc7f00b8240 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7dc004eb0 con 0x7fc7f00b8240 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1089936849 0 0) 0x7fc7f0137ad0 con 0x7fc7f0144a70 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 >> v1:172.21.15.19:6790/0 conn(0x7fc7f009c8c0 legacy=0x7fc7f01384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 >> v1:172.21.15.62:6789/0 conn(0x7fc7f00b8240 legacy=0x7fc7f0135ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 monclient: found mon.a 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0137db0 con 0x7fc7f0144a70 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:19:24.221 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 10 monclient: _renew_subs 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f0138090 con 0x7fc7f0144a70 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 10 monclient: _renew_subs 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f0144ec0 con 0x7fc7f0144a70 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7f59c4640 1 librados: init done 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7e8003970 con 0x7fc7f0144a70 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc7e8003d00 con 0x7fc7f0144a70 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.222 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 10 monclient: dump: 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:epoch 1 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc7e80042b0 con 0x7fc7f0144a70 2024-03-29T11:19:24.223 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.229+0000 7fc7ed7fa640 4 set_mon_vals no callback set 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fc7e8004560 con 0x7fc7f0144a70 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7d7fff640 10 client.4191.objecter ms_dispatch 0x7fc7f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7d7fff640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7d7fff640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7d7fff640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7f59c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7f59c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7f59c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.230+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d0005080 con 0x7fc7f0144a70 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.232+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc7e8004720 con 0x7fc7f0144a70 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.232+0000 7fc7d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.224 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.232+0000 7fc7d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:24.225 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.246+0000 7fca0e213040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:24.225 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.238+0000 7fa621645040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:19:24.262 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.285+0000 7fa621645040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:19:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7f59c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7f59c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7f59c4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:24.287 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7d0005330 con 0x7fc7f0144a70 2024-03-29T11:19:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7d7fff640 1 -- 172.21.15.19:0/1057554401 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fc7e8005be0 con 0x7fc7f0144a70 2024-03-29T11:19:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:24.288 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.311+0000 7fc7d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:24.288 INFO:teuthology.orchestra.run.smithi019.stdout: 2024-03-29T11:19:24.288 INFO:teuthology.orchestra.run.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:24.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 10 librados: watch_flush enter 2024-03-29T11:19:24.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 10 librados: watch_flush exit 2024-03-29T11:19:24.289 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 10 monclient: shutdown 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 >> v1:172.21.15.19:6789/0 conn(0x7fc7f0144a70 legacy=0x7fc7f01e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 shutdown_connections 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 >> 172.21.15.19:0/1057554401 conn(0x7fc7f0066b00 msgr2=0x7fc7f013c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 shutdown_connections 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 1 -- 172.21.15.19:0/1057554401 wait complete. 2024-03-29T11:19:24.290 INFO:teuthology.orchestra.run.smithi019.stderr:2024-03-29T11:19:24.312+0000 7fc7f59c4640 1 librados: shutdown 2024-03-29T11:19:24.299 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:19:24.300 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:19:24.348 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.371+0000 7fca0e213040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:24.379 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.402+0000 7fa621645040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:24.447 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.470+0000 7fca0e213040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:24.477 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.500+0000 7fa621645040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:19:24.607 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.631+0000 7fca0e213040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:24.638 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.661+0000 7fa621645040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:19:24.651 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.674+0000 7fca0e213040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:24.663 INFO:tasks.ceph.osd.5.smithi062.stderr:2024-03-29T11:19:24.686+0000 7f6adf80e640 -1 Falling back to public interface 2024-03-29T11:19:24.671 INFO:tasks.ceph.osd.6.smithi062.stderr:2024-03-29T11:19:24.694+0000 7f803b624640 -1 Falling back to public interface 2024-03-29T11:19:24.678 INFO:tasks.ceph.osd.7.smithi062.stderr:2024-03-29T11:19:24.701+0000 7f22770f9640 -1 Falling back to public interface 2024-03-29T11:19:24.681 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-03-29T11:19:24.704+0000 7f4aa2391640 -1 Falling back to public interface 2024-03-29T11:19:24.681 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-03-29T11:19:24.704+0000 7fd56fa0f640 -1 Falling back to public interface 2024-03-29T11:19:24.682 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-03-29T11:19:24.704+0000 7fd56e3c8640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T11:19:24.682 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-03-29T11:19:24.704+0000 7f4aa1340640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-29T11:19:24.683 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.706+0000 7fa621645040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:19:24.686 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-03-29T11:19:24.709+0000 7f0ca6815640 -1 Falling back to public interface 2024-03-29T11:19:24.686 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-03-29T11:19:24.709+0000 7f2c2ca8b640 -1 Falling back to public interface 2024-03-29T11:19:24.686 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-03-29T11:19:24.710+0000 7f2c2ba3a640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6812/0: (98) Address already in use 2024-03-29T11:19:24.691 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-03-29T11:19:24.714+0000 7f967c59a640 -1 Falling back to public interface 2024-03-29T11:19:24.813 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:24.837+0000 7fca0e213040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:24.844 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:24.867+0000 7fa621645040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:19:25.105 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:25.197 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.221+0000 7f826352b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.221+0000 7f826352b640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.221+0000 7f826352b640 10 monclient: monmap: 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:19:25.222065+0000 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:19:25.222065+0000 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:25.198 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:25.199 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:25.199 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.221+0000 7f826352b640 1 Processor -- start 2024-03-29T11:19:25.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 1 -- start start 2024-03-29T11:19:25.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient: init 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient(hunting): picked mon.noname-c con 0x7f825c1457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient(hunting): picked mon.noname-b con 0x7f825c13f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient(hunting): picked mon.noname-a con 0x7f825c0a9d70 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c0aaef0 con 0x7f825c0a9d70 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c145bd0 con 0x7f825c1457f0 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c145eb0 con 0x7f825c13f650 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f826352b640 10 monclient(hunting): authenticate will time out at 885.485716s 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f8261ac8640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f825c0a9d70 0x7f825c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:32786/0 (socket says 172.21.15.19:32786) 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.222+0000 7f8261ac8640 1 -- 172.21.15.19:0/4017845 learned_addr learned my addr 172.21.15.19:0/4017845 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:25.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534178853 0 0) 0x7f825c145eb0 con 0x7f825c13f650 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8248003520 con 0x7f825c13f650 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180908559 0 0) 0x7f825c0aaef0 con 0x7f825c0a9d70 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c145eb0 con 0x7f825c0a9d70 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076345232 0 0) 0x7f825c145bd0 con 0x7f825c1457f0 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient(hunting): my global_id is 4174 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c0aaef0 con 0x7f825c1457f0 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1858783542 0 0) 0x7f825c145eb0 con 0x7f825c0a9d70 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 >> v1:172.21.15.19:6790/0 conn(0x7f825c13f650 legacy=0x7f825c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 >> v1:172.21.15.62:6789/0 conn(0x7f825c1457f0 legacy=0x7f825c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 monclient: found mon.noname-a 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c146340 con 0x7f825c0a9d70 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f826352b640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:19:25.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8254001920 con 0x7f825c0a9d70 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f826352b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: dump: 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.212 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f826352b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8254001a80 con 0x7f825c0a9d70 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 1 -- 172.21.15.19:0/4017845 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8254001ed0 con 0x7f825c0a9d70 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f826352b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f826352b640 4 set_mon_vals no callback set 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.223+0000 7f8253fff640 10 monclient: dump: 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.213 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 10 monclient: shutdown 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 1 -- 172.21.15.19:0/4017845 >> v1:172.21.15.19:6789/0 conn(0x7f825c0a9d70 legacy=0x7f825c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 1 -- 172.21.15.19:0/4017845 shutdown_connections 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 1 -- 172.21.15.19:0/4017845 >> 172.21.15.19:0/4017845 conn(0x7f825c066a00 msgr2=0x7f825c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 1 -- 172.21.15.19:0/4017845 shutdown_connections 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 1 -- 172.21.15.19:0/4017845 wait complete. 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 10 monclient: build_initial_monmap 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.224+0000 7f826352b640 10 monclient: monmap: 2024-03-29T11:19:25.214 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 librados: starting msgr at 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 librados: starting objecter 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 Processor -- start 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 -- start start 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 librados: setting wanted keys 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 librados: calling monclient init 2024-03-29T11:19:25.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient: init 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient(hunting): picked mon.noname-b con 0x7f825c0a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient(hunting): picked mon.noname-a con 0x7f825c13f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient(hunting): picked mon.noname-c con 0x7f825c1457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c140640 con 0x7f825c13f650 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c137580 con 0x7f825c0a9d70 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825c137710 con 0x7f825c1457f0 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f826352b640 10 monclient(hunting): authenticate will time out at 885.488877s 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f8260ac6640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f825c13f650 0x7f825c134da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:32802/0 (socket says 172.21.15.19:32802) 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f8261ac8640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f825c1457f0 0x7f825c1420a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:57754/0 (socket says 172.21.15.19:57754) 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.225+0000 7f8260ac6640 1 -- 172.21.15.19:0/2976616693 learned_addr learned my addr 172.21.15.19:0/2976616693 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 client.?.objecter ms_handle_connect 0x7f825c13f650 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 client.?.objecter ms_handle_connect 0x7f825c1457f0 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 client.?.objecter ms_handle_connect 0x7f825c0a9d70 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802266221 0 0) 0x7f825c140640 con 0x7f825c13f650 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): my global_id is 4197 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8240004eb0 con 0x7f825c13f650 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072242130 0 0) 0x7f825c137710 con 0x7f825c1457f0 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c140640 con 0x7f825c1457f0 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528869536 0 0) 0x7f825c137580 con 0x7f825c0a9d70 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): my global_id is 4177 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c137710 con 0x7f825c0a9d70 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1603867327 0 0) 0x7f8240004eb0 con 0x7f825c13f650 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f825c137580 con 0x7f825c13f650 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3534862608 0 0) 0x7f825c140640 con 0x7f825c1457f0 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8240004eb0 con 0x7f825c1457f0 2024-03-29T11:19:25.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 527957327 0 0) 0x7f825c137710 con 0x7f825c0a9d70 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f825c140640 con 0x7f825c0a9d70 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f824c001a70 con 0x7f825c13f650 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:25.219 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8254001d90 con 0x7f825c1457f0 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f82580017d0 con 0x7f825c0a9d70 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1345093653 0 0) 0x7f825c137580 con 0x7f825c13f650 2024-03-29T11:19:25.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 >> v1:172.21.15.19:6790/0 conn(0x7f825c1457f0 legacy=0x7f825c1420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 >> v1:172.21.15.62:6789/0 conn(0x7f825c0a9d70 legacy=0x7f825c13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 monclient: found mon.a 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c144f00 con 0x7f825c13f650 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 5 monclient: authenticate success, global_id 4197 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 10 monclient: _renew_subs 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f825c1451b0 con 0x7f825c13f650 2024-03-29T11:19:25.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f824c001cd0 con 0x7f825c13f650 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f824c003a40 con 0x7f825c13f650 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 10 monclient: dump: 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:25.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f824c003fb0 con 0x7f825c13f650 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f82537fe640 4 set_mon_vals no callback set 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 10 monclient: _renew_subs 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f825c135690 con 0x7f825c13f650 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.226+0000 7f826352b640 1 librados: init done 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f824c0063b0 con 0x7f825c13f650 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f8251ffb640 10 client.4197.objecter ms_dispatch 0x7f825c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f8251ffb640 3 client.4197.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f8251ffb640 3 client.4197.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f8251ffb640 20 client.4197.objecter dump_active .. 0 homeless 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f826352b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f826352b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f826352b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:25.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.227+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f825c075f10 con 0x7f825c13f650 2024-03-29T11:19:25.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.229+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f824c005780 con 0x7f825c13f650 2024-03-29T11:19:25.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.229+0000 7f8251ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:25.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.229+0000 7f8251ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.291+0000 7f826352b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:25.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.291+0000 7f826352b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.291+0000 7f826352b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.291+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f825c0b3680 con 0x7f825c13f650 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.292+0000 7f8251ffb640 1 -- 172.21.15.19:0/2976616693 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f824c005a00 con 0x7f825c13f650 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.292+0000 7f8251ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.292+0000 7f8251ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:19:25.269 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:25.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 10 librados: watch_flush enter 2024-03-29T11:19:25.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 10 librados: watch_flush exit 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 20 client.4197.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 10 client.4197.objecter successfully canceled tick 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 10 monclient: shutdown 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 >> v1:172.21.15.19:6789/0 conn(0x7f825c13f650 legacy=0x7f825c134da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 shutdown_connections 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 >> 172.21.15.19:0/2976616693 conn(0x7f825c066a00 msgr2=0x7f825c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 shutdown_connections 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 1 -- 172.21.15.19:0/2976616693 wait complete. 2024-03-29T11:19:25.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:25.293+0000 7f826352b640 1 librados: shutdown 2024-03-29T11:19:25.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:25.654 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:25.678+0000 7fca0e213040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:25.692 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:25.715+0000 7fa621645040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:19:25.941 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:25.964+0000 7fca0e213040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:25.979 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:26.002+0000 7fa621645040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:19:25.994 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:26.017+0000 7fca0e213040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:26.033 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:26.056+0000 7fa621645040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:19:26.114 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:26.138+0000 7fca0e213040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:26.155 INFO:tasks.ceph.mgr.x.smithi062.stderr:2024-03-29T11:19:26.177+0000 7fa621645040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:19:27.133 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:27.156+0000 7fc9aeb70640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:19:27.133 INFO:tasks.ceph.mgr.y.smithi019.stderr:2024-03-29T11:19:27.157+0000 7fc9aeb70640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:19:28.155 INFO:tasks.ceph.osd.5.smithi062.stderr:2024-03-29T11:19:28.178+0000 7f6adf80e640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:19:28.169 INFO:tasks.ceph.osd.1.smithi019.stderr:2024-03-29T11:19:28.192+0000 7f2c2ca8b640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:19:28.197 INFO:tasks.ceph.osd.2.smithi019.stderr:2024-03-29T11:19:28.221+0000 7f0ca6815640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:19:28.210 INFO:tasks.ceph.osd.7.smithi062.stderr:2024-03-29T11:19:28.233+0000 7f22770f9640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:19:28.211 INFO:tasks.ceph.osd.3.smithi019.stderr:2024-03-29T11:19:28.234+0000 7f4aa2391640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:19:28.213 INFO:tasks.ceph.osd.0.smithi019.stderr:2024-03-29T11:19:28.237+0000 7fd56fa0f640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:19:28.218 INFO:tasks.ceph.osd.4.smithi062.stderr:2024-03-29T11:19:28.241+0000 7f967c59a640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:19:28.222 INFO:tasks.ceph.osd.6.smithi062.stderr:2024-03-29T11:19:28.245+0000 7f803b624640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:19:32.085 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:32.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.259+0000 7f153a28d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.259+0000 7f153a28d640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.259+0000 7f153a28d640 10 monclient: monmap: 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:19:32.261620+0000 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:19:32.261620+0000 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:32.238 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:32.239 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:32.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.260+0000 7f153a28d640 1 Processor -- start 2024-03-29T11:19:32.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 1 -- start start 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 10 monclient: init 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 10 monclient(hunting): picked mon.noname-b con 0x7f15340b4790 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 10 monclient(hunting): picked mon.noname-c con 0x7f15340aea90 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 10 monclient(hunting): picked mon.noname-a con 0x7f15340b39f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.261+0000 7f153a28d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340b0c60 con 0x7f15340b39f0 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.262+0000 7f153a28d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340af020 con 0x7f15340aea90 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.262+0000 7f153a28d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153413fbd0 con 0x7f15340b4790 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.262+0000 7f153a28d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.262+0000 7f153a28d640 10 monclient(hunting): authenticate will time out at 892.526037s 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.262+0000 7f1533fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f15340b4790 0x7f15340b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:58182/0 (socket says 172.21.15.19:58182) 2024-03-29T11:19:32.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.262+0000 7f1533fff640 1 -- 172.21.15.19:0/1057860611 learned_addr learned my addr 172.21.15.19:0/1057860611 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268795001 0 0) 0x7f15340af020 con 0x7f15340aea90 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient(hunting): my global_id is 4222 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c003520 con 0x7f15340aea90 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1114622811 0 0) 0x7f151c003520 con 0x7f15340aea90 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 >> v1:172.21.15.19:6790/0 conn(0x7f15340b4790 legacy=0x7f15340b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 >> v1:172.21.15.19:6789/0 conn(0x7f15340b39f0 legacy=0x7f153413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153413fd60 con 0x7f15340aea90 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1524001690 con 0x7f15340aea90 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: dump: 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:32.246 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15240018b0 con 0x7f15340aea90 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f153a28d640 5 monclient: authenticate success, global_id 4222 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f153a28d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f1532ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f153a28d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.263+0000 7f153a28d640 4 set_mon_vals no callback set 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f1532ffd640 1 -- 172.21.15.19:0/1057860611 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1524001c40 con 0x7f15340aea90 2024-03-29T11:19:32.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f1532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f1532ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f1532ffd640 10 monclient: dump: 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f153a28d640 10 monclient: shutdown 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f153a28d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.264+0000 7f153a28d640 1 -- 172.21.15.19:0/1057860611 >> v1:172.21.15.62:6789/0 conn(0x7f15340aea90 legacy=0x7f15340af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.265+0000 7f153a28d640 1 -- 172.21.15.19:0/1057860611 shutdown_connections 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.265+0000 7f153a28d640 1 -- 172.21.15.19:0/1057860611 >> 172.21.15.19:0/1057860611 conn(0x7f1534066a00 msgr2=0x7f1534066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.265+0000 7f153a28d640 1 -- 172.21.15.19:0/1057860611 shutdown_connections 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.266+0000 7f153a28d640 1 -- 172.21.15.19:0/1057860611 wait complete. 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.267+0000 7f153a28d640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.267+0000 7f153a28d640 10 monclient: monmap: 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:32.249 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:32.250 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 1 librados: starting msgr at 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 1 librados: starting objecter 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 1 Processor -- start 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 1 -- start start 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 1 librados: setting wanted keys 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 1 librados: calling monclient init 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.268+0000 7f153a28d640 10 monclient: init 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.270+0000 7f153a28d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.270+0000 7f153a28d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.270+0000 7f153a28d640 10 monclient(hunting): picked mon.noname-b con 0x7f15340b39f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.270+0000 7f1533fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f15340b39f0 0x7f153412b7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:58960/0 (socket says 172.21.15.19:58960) 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.270+0000 7f1533fff640 1 -- 172.21.15.19:0/1784297063 learned_addr learned my addr 172.21.15.19:0/1784297063 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.270+0000 7f153a28d640 10 monclient(hunting): picked mon.noname-a con 0x7f15340b4790 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 client.?.objecter ms_handle_connect 0x7f15340b39f0 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f153a28d640 10 monclient(hunting): picked mon.noname-c con 0x7f153406ec20 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153406fb20 con 0x7f15340b4790 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534070cc0 con 0x7f15340b39f0 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153412c5e0 con 0x7f153406ec20 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f153a28d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f153a28d640 10 monclient(hunting): authenticate will time out at 892.535345s 2024-03-29T11:19:32.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 client.?.objecter ms_handle_connect 0x7f15340b4790 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 client.?.objecter ms_handle_connect 0x7f153406ec20 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137293169 0 0) 0x7f1534070cc0 con 0x7f15340b39f0 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 10 monclient(hunting): my global_id is 4225 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.271+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1520004eb0 con 0x7f15340b39f0 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988395441 0 0) 0x7f153406fb20 con 0x7f15340b4790 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1534070cc0 con 0x7f15340b4790 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572879005 0 0) 0x7f153412c5e0 con 0x7f153406ec20 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1520005360 con 0x7f153406ec20 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2439789054 0 0) 0x7f1520004eb0 con 0x7f15340b39f0 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f153412c5e0 con 0x7f15340b39f0 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1528001890 con 0x7f15340b39f0 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:32.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.272+0000 7f1530ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3351716775 0 0) 0x7f1534070cc0 con 0x7f15340b4790 2024-03-29T11:19:32.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1520004eb0 con 0x7f15340b4790 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3981469791 0 0) 0x7f153412c5e0 con 0x7f15340b39f0 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 >> v1:172.21.15.19:6790/0 conn(0x7f153406ec20 legacy=0x7f153413df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 >> v1:172.21.15.19:6789/0 conn(0x7f15340b4790 legacy=0x7f153412bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 monclient: found mon.b 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1534070fa0 con 0x7f15340b39f0 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f1530ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 5 monclient: authenticate success, global_id 4225 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 10 monclient: _renew_subs 2024-03-29T11:19:32.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153413e670 con 0x7f15340b39f0 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 10 monclient: _renew_subs 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15341dc9d0 con 0x7f15340b39f0 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.273+0000 7f153a28d640 1 librados: init done 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15280047e0 con 0x7f15340b39f0 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f1530ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1528001d80 con 0x7f15340b39f0 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f1530ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f1530ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f1530ff9640 10 monclient: dump: 2024-03-29T11:19:32.260 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.274+0000 7f15327fc640 4 set_mon_vals no callback set 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.275+0000 7f153a28d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.275+0000 7f153a28d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.275+0000 7f153a28d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:32.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.275+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1514005080 con 0x7f15340b39f0 2024-03-29T11:19:32.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.276+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f1528004bf0 con 0x7f15340b39f0 2024-03-29T11:19:32.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.277+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f1528084a80 con 0x7f15340b39f0 2024-03-29T11:19:32.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.277+0000 7f1530ff9640 10 client.4225.objecter ms_dispatch 0x7f1534000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:32.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.277+0000 7f1530ff9640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.277+0000 7f1530ff9640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.277+0000 7f1530ff9640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.277+0000 7f15337fe640 10 monclient: get_auth_request con 0x7f152006e030 auth_method 0 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.278+0000 7f1530ff9640 10 client.4225.objecter ms_handle_connect 0x7f152006e030 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.285+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1528050360 con 0x7f15340b39f0 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.285+0000 7f1530ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.285+0000 7f1530ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:32.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.696+0000 7f153a28d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:32.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.696+0000 7f153a28d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:32.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.696+0000 7f153a28d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:32.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.696+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1514005330 con 0x7f15340b39f0 2024-03-29T11:19:32.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.697+0000 7f1530ff9640 1 -- 172.21.15.19:0/1784297063 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f1528052290 con 0x7f15340b39f0 2024-03-29T11:19:32.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.697+0000 7f1530ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:32.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.697+0000 7f1530ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:32.676 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:19:32.676 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:32.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 10 librados: watch_flush enter 2024-03-29T11:19:32.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 10 librados: watch_flush exit 2024-03-29T11:19:32.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-29T11:19:32.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 10 client.4225.objecter successfully canceled tick 2024-03-29T11:19:32.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 >> v1:172.21.15.19:6816/3721232097 conn(0x7f152006e030 legacy=0x7f15200704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 10 monclient: shutdown 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 >> v1:172.21.15.62:6789/0 conn(0x7f15340b39f0 legacy=0x7f153412b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.701+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 shutdown_connections 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.702+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 >> 172.21.15.19:0/1784297063 conn(0x7f1534066a00 msgr2=0x7f15340b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.702+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 shutdown_connections 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.702+0000 7f153a28d640 1 -- 172.21.15.19:0/1784297063 wait complete. 2024-03-29T11:19:32.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:32.702+0000 7f153a28d640 1 librados: shutdown 2024-03-29T11:19:32.694 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:39.500 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:39.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.676+0000 7f876576d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:39.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.676+0000 7f876576d640 10 monclient: build_initial_monmap 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.677+0000 7f876576d640 10 monclient: monmap: 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:19:39.678769+0000 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:19:39.678769+0000 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:39.655 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:39.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.677+0000 7f876576d640 1 Processor -- start 2024-03-29T11:19:39.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.678+0000 7f876576d640 1 -- start start 2024-03-29T11:19:39.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.678+0000 7f876576d640 10 monclient: init 2024-03-29T11:19:39.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:39.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:39.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient(hunting): picked mon.noname-b con 0x7f87600b0770 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:39.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient(hunting): picked mon.noname-c con 0x7f876014a700 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:39.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f875effd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f87600b0770 0x7f8760149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:55384/0 (socket says 172.21.15.19:55384) 2024-03-29T11:19:39.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient(hunting): picked mon.noname-a con 0x7f87600ac5a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f875f7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f87600ac5a0 0x7f8760151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53236/0 (socket says 172.21.15.19:53236) 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f875f7fe640 1 -- 172.21.15.19:0/4132864634 learned_addr learned my addr 172.21.15.19:0/4132864634 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600aba70 con 0x7f87600ac5a0 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600aecb0 con 0x7f876014a700 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600af150 con 0x7f87600b0770 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.679+0000 7f876576d640 10 monclient(hunting): authenticate will time out at 899.943702s 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1057444662 0 0) 0x7f87600af150 con 0x7f87600b0770 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748003520 con 0x7f87600b0770 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769616021 0 0) 0x7f87600aba70 con 0x7f87600ac5a0 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87600af150 con 0x7f87600ac5a0 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243995406 0 0) 0x7f87600aecb0 con 0x7f876014a700 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient(hunting): my global_id is 4228 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87600aba70 con 0x7f876014a700 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4098923212 0 0) 0x7f8748003520 con 0x7f87600b0770 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 >> v1:172.21.15.62:6789/0 conn(0x7f876014a700 legacy=0x7f87600abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 >> v1:172.21.15.19:6789/0 conn(0x7f87600ac5a0 legacy=0x7f8760151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 monclient: found mon.noname-b 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600ac9d0 con 0x7f87600b0770 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:39.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f876576d640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f876576d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8750001a10 con 0x7f87600b0770 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f875dffb640 10 monclient: dump: 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:39.662 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.680+0000 7f876576d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8750001c30 con 0x7f87600b0770 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f875dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f876576d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f876576d640 4 set_mon_vals no callback set 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f875dffb640 1 -- 172.21.15.19:0/4132864634 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8750001fc0 con 0x7f87600b0770 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f875dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f875dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f875dffb640 10 monclient: dump: 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:39.663 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f876576d640 10 monclient: shutdown 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f876576d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:39.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.681+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 >> v1:172.21.15.19:6790/0 conn(0x7f87600b0770 legacy=0x7f8760149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.682+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 shutdown_connections 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.682+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 >> 172.21.15.19:0/4132864634 conn(0x7f876006b660 msgr2=0x7f876006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.682+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 shutdown_connections 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.682+0000 7f876576d640 1 -- 172.21.15.19:0/4132864634 wait complete. 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.683+0000 7f876576d640 10 monclient: build_initial_monmap 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.683+0000 7f876576d640 10 monclient: monmap: 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:39.669 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.683+0000 7f876576d640 1 librados: starting msgr at 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.683+0000 7f876576d640 1 librados: starting objecter 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.684+0000 7f876576d640 1 Processor -- start 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.684+0000 7f876576d640 1 -- start start 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.684+0000 7f876576d640 1 librados: setting wanted keys 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.684+0000 7f876576d640 1 librados: calling monclient init 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.684+0000 7f876576d640 10 monclient: init 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f876576d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f876576d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:39.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f876576d640 10 monclient(hunting): picked mon.noname-a con 0x7f87600ac5a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f875effd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f87600ac5a0 0x7f8760146f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:54648/0 (socket says 172.21.15.19:54648) 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f875effd640 1 -- 172.21.15.19:0/1730817824 learned_addr learned my addr 172.21.15.19:0/1730817824 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f876576d640 10 monclient(hunting): picked mon.noname-c con 0x7f87600b0770 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f876576d640 10 monclient(hunting): picked mon.noname-b con 0x7f876014a700 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.685+0000 7f87477fe640 10 client.?.objecter ms_handle_connect 0x7f87600ac5a0 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8760125530 con 0x7f87600ac5a0 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8760128490 con 0x7f876014a700 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f876012a540 con 0x7f87600b0770 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 client.?.objecter ms_handle_connect 0x7f87600b0770 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f876576d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f876576d640 10 monclient(hunting): authenticate will time out at 899.950313s 2024-03-29T11:19:39.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811790528 0 0) 0x7f8760125530 con 0x7f87600ac5a0 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f874c004eb0 con 0x7f87600ac5a0 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860498058 0 0) 0x7f876012a540 con 0x7f87600b0770 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8760125530 con 0x7f87600b0770 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159213471 0 0) 0x7f874c004eb0 con 0x7f87600ac5a0 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f876012a540 con 0x7f87600ac5a0 2024-03-29T11:19:39.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8750001a10 con 0x7f87600ac5a0 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): dump: 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:39.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.686+0000 7f87477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 client.?.objecter ms_handle_connect 0x7f876014a700 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3236557001 0 0) 0x7f876012a540 con 0x7f87600ac5a0 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 >> v1:172.21.15.19:6790/0 conn(0x7f87600b0770 legacy=0x7f8760128920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 >> v1:172.21.15.62:6789/0 conn(0x7f876014a700 legacy=0x7f87601e70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 monclient: found mon.a 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8760127ee0 con 0x7f87600ac5a0 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:19:39.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 10 monclient: _renew_subs 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8760128190 con 0x7f87600ac5a0 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 10 monclient: _renew_subs 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87601dfb90 con 0x7f87600ac5a0 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f876576d640 1 librados: init done 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8750001570 con 0x7f87600ac5a0 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8750005070 con 0x7f87600ac5a0 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f87477fe640 10 monclient: dump: 2024-03-29T11:19:39.675 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.687+0000 7f875d7fa640 4 set_mon_vals no callback set 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.689+0000 7f876576d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.689+0000 7f876576d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.689+0000 7f876576d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.689+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8740005080 con 0x7f87600ac5a0 2024-03-29T11:19:39.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.689+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f87500051d0 con 0x7f87600ac5a0 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.690+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f8750084fb0 con 0x7f87600ac5a0 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.690+0000 7f87477fe640 10 client.4245.objecter ms_dispatch 0x7f8760000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.690+0000 7f87477fe640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.690+0000 7f87477fe640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.690+0000 7f87477fe640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.690+0000 7f875e7fc640 10 monclient: get_auth_request con 0x7f874c06dd90 auth_method 0 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.691+0000 7f87477fe640 10 client.4245.objecter ms_handle_connect 0x7f874c06dd90 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.697+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87500506c0 con 0x7f87600ac5a0 2024-03-29T11:19:39.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.698+0000 7f87477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:39.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:39.698+0000 7f87477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:40.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.068+0000 7f876576d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:40.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.068+0000 7f876576d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:40.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.068+0000 7f876576d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:40.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.068+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8740005330 con 0x7f87600ac5a0 2024-03-29T11:19:40.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.070+0000 7f87477fe640 1 -- 172.21.15.19:0/1730817824 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f87500527c0 con 0x7f87600ac5a0 2024-03-29T11:19:40.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.070+0000 7f87477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:40.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.070+0000 7f87477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:40.048 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:19:40.048 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:40.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.072+0000 7f876576d640 10 librados: watch_flush enter 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.072+0000 7f876576d640 10 librados: watch_flush exit 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.072+0000 7f876576d640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.072+0000 7f876576d640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.072+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 >> v1:172.21.15.19:6816/3721232097 conn(0x7f874c06dd90 legacy=0x7f874c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 10 monclient: shutdown 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 >> v1:172.21.15.19:6789/0 conn(0x7f87600ac5a0 legacy=0x7f8760146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 shutdown_connections 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 >> 172.21.15.19:0/1730817824 conn(0x7f876006b660 msgr2=0x7f87600abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 shutdown_connections 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 1 -- 172.21.15.19:0/1730817824 wait complete. 2024-03-29T11:19:40.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:40.073+0000 7f876576d640 1 librados: shutdown 2024-03-29T11:19:40.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:46.872 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.039+0000 7f1c8d2a2640 10 monclient: get_monmap_and_config 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.039+0000 7f1c8d2a2640 10 monclient: build_initial_monmap 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.039+0000 7f1c8d2a2640 10 monclient: monmap: 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:19:47.041689+0000 2024-03-29T11:19:47.019 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:19:47.041689+0000 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.041+0000 7f1c8d2a2640 1 Processor -- start 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.041+0000 7f1c8d2a2640 1 -- start start 2024-03-29T11:19:47.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.041+0000 7f1c8d2a2640 10 monclient: init 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.042+0000 7f1c8d2a2640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.042+0000 7f1c8d2a2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.042+0000 7f1c8d2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f1c80034850 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.042+0000 7f1c8d2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8002a640 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.043+0000 7f1c8d2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f1c800327d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.043+0000 7f1c8d2a2640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8002c820 con 0x7f1c8002a640 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.043+0000 7f1c87fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1c80034850 0x7f1c800c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:33670/0 (socket says 172.21.15.19:33670) 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.043+0000 7f1c8d2a2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8002a120 con 0x7f1c80034850 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.043+0000 7f1c87fff640 1 -- 172.21.15.19:0/53154181 learned_addr learned my addr 172.21.15.19:0/53154181 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:47.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.044+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/53154181 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800c75b0 con 0x7f1c800327d0 2024-03-29T11:19:47.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.044+0000 7f1c8d2a2640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:47.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.044+0000 7f1c8d2a2640 10 monclient(hunting): authenticate will time out at 907.308021s 2024-03-29T11:19:47.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548637674 0 0) 0x7f1c8002c820 con 0x7f1c8002a640 2024-03-29T11:19:47.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:47.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c78003520 con 0x7f1c8002a640 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252164697 0 0) 0x7f1c800c75b0 con 0x7f1c800327d0 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c780048a0 con 0x7f1c800327d0 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561536439 0 0) 0x7f1c8002a120 con 0x7f1c80034850 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:47.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c78004be0 con 0x7f1c80034850 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103011089 0 0) 0x7f1c780048a0 con 0x7f1c800327d0 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 >> v1:172.21.15.62:6789/0 conn(0x7f1c80034850 legacy=0x7f1c800c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 >> v1:172.21.15.19:6789/0 conn(0x7f1c8002a640 legacy=0x7f1c800320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 monclient: found mon.noname-b 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c80033110 con 0x7f1c800327d0 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:47.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c8d2a2640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c8d2a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.046+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1c7c001670 con 0x7f1c800327d0 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c86ffd640 10 monclient: dump: 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.029 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c8d2a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c7c001850 con 0x7f1c800327d0 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c8d2a2640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.047+0000 7f1c8d2a2640 4 set_mon_vals no callback set 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c8d2a2640 10 monclient: shutdown 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c86ffd640 1 -- 172.21.15.19:0/53154181 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1c7c001670 con 0x7f1c800327d0 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:47.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c86ffd640 10 monclient: dump: 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c8d2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.048+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/53154181 >> v1:172.21.15.19:6790/0 conn(0x7f1c800327d0 legacy=0x7f1c800cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.049+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/53154181 shutdown_connections 2024-03-29T11:19:47.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.049+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/53154181 >> 172.21.15.19:0/53154181 conn(0x7f1c800058f0 msgr2=0x7f1c80005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.049+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/53154181 shutdown_connections 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.049+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/53154181 wait complete. 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.050+0000 7f1c8d2a2640 10 monclient: build_initial_monmap 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.050+0000 7f1c8d2a2640 10 monclient: monmap: 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:47.036 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 1 librados: starting msgr at 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 1 librados: starting objecter 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 1 Processor -- start 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 1 -- start start 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 1 librados: setting wanted keys 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 1 librados: calling monclient init 2024-03-29T11:19:47.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.051+0000 7f1c8d2a2640 10 monclient: init 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.052+0000 7f1c8d2a2640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.052+0000 7f1c8d2a2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8002a640 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 10 monclient(hunting): picked mon.noname-c con 0x7f1c800327d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 10 monclient(hunting): picked mon.noname-b con 0x7f1c80034850 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8002d130 con 0x7f1c8002a640 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800c0290 con 0x7f1c80034850 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c800bff00 con 0x7f1c800327d0 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.053+0000 7f1c8d2a2640 10 monclient(hunting): authenticate will time out at 907.317488s 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.055+0000 7f1c8caa1640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f1c80034850 0x7f1c8015c1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:33872/0 (socket says 172.21.15.19:33872) 2024-03-29T11:19:47.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.055+0000 7f1c8caa1640 1 -- 172.21.15.19:0/3278963109 learned_addr learned my addr 172.21.15.19:0/3278963109 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 10 client.?.objecter ms_handle_connect 0x7f1c8002a640 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222687634 0 0) 0x7f1c8002d130 con 0x7f1c8002a640 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.056+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c004eb0 con 0x7f1c8002a640 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2021605575 0 0) 0x7f1c6c004eb0 con 0x7f1c8002a640 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c8002d130 con 0x7f1c8002a640 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1c74001070 con 0x7f1c8002a640 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:47.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.057+0000 7f1c84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.058+0000 7f1c84ff9640 10 client.?.objecter ms_handle_connect 0x7f1c80034850 2024-03-29T11:19:47.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.058+0000 7f1c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.058+0000 7f1c84ff9640 10 client.?.objecter ms_handle_connect 0x7f1c800327d0 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.058+0000 7f1c84ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341861109 0 0) 0x7f1c800bff00 con 0x7f1c800327d0 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:19:47.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c004eb0 con 0x7f1c800327d0 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1038954115 0 0) 0x7f1c6c004eb0 con 0x7f1c800327d0 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c800bff00 con 0x7f1c800327d0 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1c8804fdd0 con 0x7f1c800327d0 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:47.044 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4263107237 0 0) 0x7f1c8002d130 con 0x7f1c8002a640 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 >> v1:172.21.15.19:6790/0 conn(0x7f1c800327d0 legacy=0x7f1c8015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 >> v1:172.21.15.62:6789/0 conn(0x7f1c80034850 legacy=0x7f1c8015c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 monclient: found mon.a 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800c3ac0 con 0x7f1c8002a640 2024-03-29T11:19:47.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c740015e0 con 0x7f1c8002a640 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1c740019b0 con 0x7f1c8002a640 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c84ff9640 10 monclient: dump: 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:47.046 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 10 monclient: _renew_subs 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c867fc640 4 set_mon_vals no callback set 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c800c3d70 con 0x7f1c8002a640 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 10 monclient: _renew_subs 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c801641e0 con 0x7f1c8002a640 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.059+0000 7f1c8d2a2640 1 librados: init done 2024-03-29T11:19:47.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.061+0000 7f1c8d2a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.061+0000 7f1c8d2a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.061+0000 7f1c8d2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.061+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c800c04d0 con 0x7f1c8002a640 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.063+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f1c74001b10 con 0x7f1c8002a640 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.063+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f1c740849b0 con 0x7f1c8002a640 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.063+0000 7f1c84ff9640 10 client.4251.objecter ms_dispatch 0x7f1c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.063+0000 7f1c84ff9640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.063+0000 7f1c84ff9640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.064+0000 7f1c84ff9640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.064+0000 7f1c877fe640 10 monclient: get_auth_request con 0x7f1c6c06dd30 auth_method 0 2024-03-29T11:19:47.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.064+0000 7f1c84ff9640 10 client.4251.objecter ms_handle_connect 0x7f1c6c06dd30 2024-03-29T11:19:47.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.072+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c800c04d0 con 0x7f1c8002a640 2024-03-29T11:19:47.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.072+0000 7f1c84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:47.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.072+0000 7f1c84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:47.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.404+0000 7f1c8d2a2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:47.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.404+0000 7f1c8d2a2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:47.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.404+0000 7f1c8d2a2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:19:47.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.404+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c800c04d0 con 0x7f1c8002a640 2024-03-29T11:19:47.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.405+0000 7f1c84ff9640 1 -- 172.21.15.19:0/3278963109 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f1c800c04d0 con 0x7f1c8002a640 2024-03-29T11:19:47.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.405+0000 7f1c84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:47.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.405+0000 7f1c84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:47.383 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:19:47.383 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.408+0000 7f1c8d2a2640 10 librados: watch_flush enter 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.408+0000 7f1c8d2a2640 10 librados: watch_flush exit 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 >> v1:172.21.15.19:6816/3721232097 conn(0x7f1c6c06dd30 legacy=0x7f1c6c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 10 monclient: shutdown 2024-03-29T11:19:47.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:47.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 >> v1:172.21.15.19:6789/0 conn(0x7f1c8002a640 legacy=0x7f1c800bf7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 shutdown_connections 2024-03-29T11:19:47.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 >> 172.21.15.19:0/3278963109 conn(0x7f1c800058f0 msgr2=0x7f1c800043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:47.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 shutdown_connections 2024-03-29T11:19:47.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 -- 172.21.15.19:0/3278963109 wait complete. 2024-03-29T11:19:47.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:47.409+0000 7f1c8d2a2640 1 librados: shutdown 2024-03-29T11:19:47.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:54.213 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.381+0000 7f6332164640 10 monclient: get_monmap_and_config 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.381+0000 7f6332164640 10 monclient: build_initial_monmap 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.381+0000 7f6332164640 10 monclient: monmap: 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:19:54.383074+0000 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:19:54.383074+0000 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:19:54.360 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:19:54.360 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:54.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.382+0000 7f6332164640 1 Processor -- start 2024-03-29T11:19:54.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.382+0000 7f6332164640 1 -- start start 2024-03-29T11:19:54.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.382+0000 7f6332164640 10 monclient: init 2024-03-29T11:19:54.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.383+0000 7f6332164640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:54.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.383+0000 7f6332164640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:54.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 10 monclient(hunting): picked mon.noname-b con 0x7f6324006b60 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:54.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 10 monclient(hunting): picked mon.noname-c con 0x7f632402c5c0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 10 monclient(hunting): picked mon.noname-a con 0x7f63240c2c20 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632402be50 con 0x7f63240c2c20 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632402c130 con 0x7f632402c5c0 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632402e310 con 0x7f6324006b60 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.384+0000 7f6332164640 10 monclient(hunting): authenticate will time out at 914.648546s 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.385+0000 7f6330961640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f632402c5c0 0x7f63240c2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:33340/0 (socket says 172.21.15.19:33340) 2024-03-29T11:19:54.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.385+0000 7f6330961640 1 -- 172.21.15.19:0/414417301 learned_addr learned my addr 172.21.15.19:0/414417301 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:54.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822883585 0 0) 0x7f632402c130 con 0x7f632402c5c0 2024-03-29T11:19:54.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:54.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:54.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:54.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:19:54.365 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6318003520 con 0x7f632402c5c0 2024-03-29T11:19:54.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1605430386 0 0) 0x7f6318003520 con 0x7f632402c5c0 2024-03-29T11:19:54.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 >> v1:172.21.15.19:6790/0 conn(0x7f6324006b60 legacy=0x7f63240c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:54.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 >> v1:172.21.15.19:6789/0 conn(0x7f63240c2c20 legacy=0x7f63240d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:54.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:54.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 monclient: found mon.noname-c 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632402e5f0 con 0x7f632402c5c0 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f632c04f730 con 0x7f632402c5c0 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6323fff640 10 monclient: dump: 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:54.367 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6332164640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.388+0000 7f6332164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:54.368 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.389+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f632c075f10 con 0x7f632402c5c0 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.389+0000 7f6323fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.389+0000 7f6332164640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.389+0000 7f6332164640 4 set_mon_vals no callback set 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6332164640 10 monclient: shutdown 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6332164640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6332164640 1 -- 172.21.15.19:0/414417301 >> v1:172.21.15.62:6789/0 conn(0x7f632402c5c0 legacy=0x7f63240c2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6323fff640 1 -- 172.21.15.19:0/414417301 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f632c0758b0 con 0x7f632402c5c0 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6323fff640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6332164640 1 -- 172.21.15.19:0/414417301 shutdown_connections 2024-03-29T11:19:54.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.390+0000 7f6332164640 1 -- 172.21.15.19:0/414417301 >> 172.21.15.19:0/414417301 conn(0x7f63240058f0 msgr2=0x7f6324005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:54.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.391+0000 7f6332164640 1 -- 172.21.15.19:0/414417301 shutdown_connections 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.391+0000 7f6332164640 1 -- 172.21.15.19:0/414417301 wait complete. 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.392+0000 7f6332164640 10 monclient: build_initial_monmap 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.392+0000 7f6332164640 10 monclient: monmap: 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:19:54.372 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.392+0000 7f6332164640 1 librados: starting msgr at 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.393+0000 7f6332164640 1 librados: starting objecter 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.393+0000 7f6332164640 1 Processor -- start 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.393+0000 7f6332164640 1 -- start start 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.393+0000 7f6332164640 1 librados: setting wanted keys 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.393+0000 7f6332164640 1 librados: calling monclient init 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.393+0000 7f6332164640 10 monclient: init 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.394+0000 7f6332164640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:54.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.394+0000 7f6332164640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 10 monclient(hunting): picked mon.noname-a con 0x7f6324006b60 addr v1:172.21.15.19:6789/0 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 10 monclient(hunting): picked mon.noname-b con 0x7f632402c5c0 addr v1:172.21.15.62:6789/0 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 10 monclient(hunting): picked mon.noname-c con 0x7f63240c2c20 addr v1:172.21.15.19:6790/0 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6324030f40 con 0x7f6324006b60 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240c4000 con 0x7f632402c5c0 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63240bc690 con 0x7f63240c2c20 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6332164640 10 monclient(hunting): authenticate will time out at 914.660382s 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6330961640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f632402c5c0 0x7f632402f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:33736/0 (socket says 172.21.15.19:33736) 2024-03-29T11:19:54.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.396+0000 7f6330961640 1 -- 172.21.15.19:0/1480148487 learned_addr learned my addr 172.21.15.19:0/1480148487 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:19:54.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 client.?.objecter ms_handle_connect 0x7f632402c5c0 2024-03-29T11:19:54.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:54.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193073145 0 0) 0x7f63240c4000 con 0x7f632402c5c0 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6310004eb0 con 0x7f632402c5c0 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 client.?.objecter ms_handle_connect 0x7f6324006b60 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554033458 0 0) 0x7f6310004eb0 con 0x7f632402c5c0 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.397+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63240c4000 con 0x7f632402c5c0 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f632c068890 con 0x7f632402c5c0 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 10 monclient(hunting): dump: 2024-03-29T11:19:54.376 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3344673859 0 0) 0x7f63240c4000 con 0x7f632402c5c0 2024-03-29T11:19:54.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.398+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 >> v1:172.21.15.19:6790/0 conn(0x7f63240c2c20 legacy=0x7f632402fb90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:54.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 >> v1:172.21.15.19:6789/0 conn(0x7f6324006b60 legacy=0x7f63240bbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:54.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 1 monclient: found mon.b 2024-03-29T11:19:54.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:54.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:54.379 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63240bc820 con 0x7f632402c5c0 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6321ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6332164640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6332164640 10 monclient: _renew_subs 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6332164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.399+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63240bca70 con 0x7f632402c5c0 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6332164640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6332164640 10 monclient: _renew_subs 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6332164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240c4700 con 0x7f632402c5c0 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6332164640 1 librados: init done 2024-03-29T11:19:54.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f632c04fdd0 con 0x7f632402c5c0 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f6321ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.400+0000 7f63237fe640 4 set_mon_vals no callback set 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f632c067bf0 con 0x7f632402c5c0 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6321ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6321ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6321ffb640 10 monclient: dump: 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:19:54.381 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:19:54.382 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:19:54.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6332164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:54.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6332164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:54.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6332164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:54.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.401+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c005080 con 0x7f632402c5c0 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.403+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f632c067d50 con 0x7f632402c5c0 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.403+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f632c0ee860 con 0x7f632402c5c0 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.403+0000 7f6321ffb640 10 client.4237.objecter ms_dispatch 0x7f632c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.403+0000 7f6321ffb640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.403+0000 7f6321ffb640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.403+0000 7f6321ffb640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.404+0000 7f6331162640 10 monclient: get_auth_request con 0x7f631006da10 auth_method 0 2024-03-29T11:19:54.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.404+0000 7f6321ffb640 10 client.4237.objecter ms_handle_connect 0x7f631006da10 2024-03-29T11:19:54.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.411+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f632c0ba140 con 0x7f632402c5c0 2024-03-29T11:19:54.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.411+0000 7f6321ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:54.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.411+0000 7f6321ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:54.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.734+0000 7f6332164640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:54.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.734+0000 7f6332164640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:54.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.734+0000 7f6332164640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:19:54.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.734+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f630c005330 con 0x7f632402c5c0 2024-03-29T11:19:54.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.735+0000 7f6321ffb640 1 -- 172.21.15.19:0/1480148487 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f632c0bc070 con 0x7f632402c5c0 2024-03-29T11:19:54.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.735+0000 7f6321ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:54.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.735+0000 7f6321ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:54.714 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:19:54.714 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:54.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.739+0000 7f6332164640 10 librados: watch_flush enter 2024-03-29T11:19:54.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.739+0000 7f6332164640 10 librados: watch_flush exit 2024-03-29T11:19:54.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.739+0000 7f6332164640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T11:19:54.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.739+0000 7f6332164640 10 client.4237.objecter successfully canceled tick 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.739+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 >> v1:172.21.15.19:6816/3721232097 conn(0x7f631006da10 legacy=0x7f631006fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 10 monclient: shutdown 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 >> v1:172.21.15.62:6789/0 conn(0x7f632402c5c0 legacy=0x7f632402f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 shutdown_connections 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 >> 172.21.15.19:0/1480148487 conn(0x7f63240058f0 msgr2=0x7f63240c22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 shutdown_connections 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 1 -- 172.21.15.19:0/1480148487 wait complete. 2024-03-29T11:19:54.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:19:54.740+0000 7f6332164640 1 librados: shutdown 2024-03-29T11:19:54.733 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:01.540 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:01.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.714+0000 7ff06d472640 10 monclient: get_monmap_and_config 2024-03-29T11:20:01.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.714+0000 7ff06d472640 10 monclient: build_initial_monmap 2024-03-29T11:20:01.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.714+0000 7ff06d472640 10 monclient: monmap: 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:01.714922+0000 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:01.714922+0000 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:01.691 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:01.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.715+0000 7ff06d472640 1 Processor -- start 2024-03-29T11:20:01.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.715+0000 7ff06d472640 1 -- start start 2024-03-29T11:20:01.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.715+0000 7ff06d472640 10 monclient: init 2024-03-29T11:20:01.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.716+0000 7ff06d472640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:01.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.716+0000 7ff06d472640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:01.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.716+0000 7ff06d472640 10 monclient(hunting): picked mon.noname-c con 0x7ff0680b1f60 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:01.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.716+0000 7ff06d472640 10 monclient(hunting): picked mon.noname-b con 0x7ff068071eb0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:01.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.719+0000 7ff06d472640 10 monclient(hunting): picked mon.noname-a con 0x7ff068143d10 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.719+0000 7ff06d472640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06806ef80 con 0x7ff068143d10 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.719+0000 7ff06d472640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06806f260 con 0x7ff0680b1f60 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.719+0000 7ff06d472640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680b40c0 con 0x7ff068071eb0 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.719+0000 7ff06d472640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.719+0000 7ff06d472640 10 monclient(hunting): authenticate will time out at 921.982899s 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.721+0000 7ff0667fc640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff068071eb0 0x7ff068146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48491/0 (socket says 172.21.15.19:48491) 2024-03-29T11:20:01.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.721+0000 7ff0667fc640 1 -- 172.21.15.19:0/951380847 learned_addr learned my addr 172.21.15.19:0/951380847 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639567299 0 0) 0x7ff0680b40c0 con 0x7ff068071eb0 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff050003520 con 0x7ff068071eb0 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438036868 0 0) 0x7ff06806ef80 con 0x7ff068143d10 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0680b40c0 con 0x7ff068143d10 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3248497442 0 0) 0x7ff050003520 con 0x7ff068071eb0 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 >> v1:172.21.15.62:6789/0 conn(0x7ff0680b1f60 legacy=0x7ff068143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 >> v1:172.21.15.19:6789/0 conn(0x7ff068143d10 legacy=0x7ff068146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 monclient: found mon.noname-b 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff068155d70 con 0x7ff068071eb0 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff058001690 con 0x7ff068071eb0 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.724+0000 7ff065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 10 monclient: dump: 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0580017f0 con 0x7ff068071eb0 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:01.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 1 -- 172.21.15.19:0/951380847 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff058001c00 con 0x7ff068071eb0 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.725+0000 7ff065ffb640 10 monclient: dump: 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:01.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.727+0000 7ff06d472640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:20:01.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.727+0000 7ff06d472640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:01.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.727+0000 7ff06d472640 4 set_mon_vals no callback set 2024-03-29T11:20:01.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.728+0000 7ff06d472640 10 monclient: shutdown 2024-03-29T11:20:01.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.728+0000 7ff06d472640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:01.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.728+0000 7ff06d472640 1 -- 172.21.15.19:0/951380847 >> v1:172.21.15.19:6790/0 conn(0x7ff068071eb0 legacy=0x7ff068146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.731+0000 7ff06d472640 1 -- 172.21.15.19:0/951380847 shutdown_connections 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.731+0000 7ff06d472640 1 -- 172.21.15.19:0/951380847 >> 172.21.15.19:0/951380847 conn(0x7ff068066a00 msgr2=0x7ff068066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.731+0000 7ff06d472640 1 -- 172.21.15.19:0/951380847 shutdown_connections 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.731+0000 7ff06d472640 1 -- 172.21.15.19:0/951380847 wait complete. 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.732+0000 7ff06d472640 10 monclient: build_initial_monmap 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.732+0000 7ff06d472640 10 monclient: monmap: 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:01.713 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 1 librados: starting msgr at 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 1 librados: starting objecter 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 1 Processor -- start 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 1 -- start start 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 1 librados: setting wanted keys 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 1 librados: calling monclient init 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.733+0000 7ff06d472640 10 monclient: init 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient(hunting): picked mon.noname-c con 0x7ff0680b1f60 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient(hunting): picked mon.noname-b con 0x7ff068143d10 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:01.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient(hunting): picked mon.noname-a con 0x7ff0681493b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:01.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff068149d20 con 0x7ff0681493b0 2024-03-29T11:20:01.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06814b000 con 0x7ff068143d10 2024-03-29T11:20:01.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06813bb20 con 0x7ff0680b1f60 2024-03-29T11:20:01.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:01.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.734+0000 7ff06d472640 10 monclient(hunting): authenticate will time out at 921.997597s 2024-03-29T11:20:01.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.740+0000 7ff0667fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ff068143d10 0x7ff06813ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:54407/0 (socket says 172.21.15.19:54407) 2024-03-29T11:20:01.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.740+0000 7ff0667fc640 1 -- 172.21.15.19:0/3971006732 learned_addr learned my addr 172.21.15.19:0/3971006732 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:01.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.740+0000 7ff04f7fe640 10 client.?.objecter ms_handle_connect 0x7ff068143d10 2024-03-29T11:20:01.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.740+0000 7ff04f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:01.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.742+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901070063 0 0) 0x7ff06814b000 con 0x7ff068143d10 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.742+0000 7ff04f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.742+0000 7ff04f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.742+0000 7ff04f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.742+0000 7ff04f7fe640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.742+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff054004eb0 con 0x7ff068143d10 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 client.?.objecter ms_handle_connect 0x7ff0680b1f60 2024-03-29T11:20:01.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:01.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 client.?.objecter ms_handle_connect 0x7ff0681493b0 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042188358 0 0) 0x7ff054004eb0 con 0x7ff068143d10 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff06814b000 con 0x7ff068143d10 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082640367 0 0) 0x7ff068149d20 con 0x7ff0681493b0 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.743+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff054004eb0 con 0x7ff0681493b0 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff058001c40 con 0x7ff068143d10 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:01.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1798070210 0 0) 0x7ff06814b000 con 0x7ff068143d10 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 >> v1:172.21.15.19:6790/0 conn(0x7ff0680b1f60 legacy=0x7ff068147400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 >> v1:172.21.15.19:6789/0 conn(0x7ff0681493b0 legacy=0x7ff06813b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 1 monclient: found mon.b 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.744+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06814b270 con 0x7ff068143d10 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.745+0000 7ff04f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.745+0000 7ff04f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:01.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.745+0000 7ff04f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 5 monclient: authenticate success, global_id 4240 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 10 monclient: _renew_subs 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff068149920 con 0x7ff068143d10 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 10 monclient: _renew_subs 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06814a1b0 con 0x7ff068143d10 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.746+0000 7ff06d472640 1 librados: init done 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.748+0000 7ff04d7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.748+0000 7ff04d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:01.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.748+0000 7ff04d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.748+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff068149b10 con 0x7ff068143d10 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff058004a90 con 0x7ff068143d10 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff04f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff058004e20 con 0x7ff068143d10 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff04f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff04f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff04f7fe640 10 monclient: dump: 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:01.726 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:01.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.749+0000 7ff064ff9640 4 set_mon_vals no callback set 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.756+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff05801adf0 con 0x7ff068143d10 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff058084bf0 con 0x7ff068143d10 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff04f7fe640 10 client.4240.objecter ms_dispatch 0x7ff068000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff066ffd640 10 monclient: get_auth_request con 0x7ff05406db90 auth_method 0 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff04f7fe640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff04f7fe640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff04f7fe640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T11:20:01.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.757+0000 7ff04f7fe640 10 client.4240.objecter ms_handle_connect 0x7ff05406db90 2024-03-29T11:20:01.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.761+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0580504d0 con 0x7ff068143d10 2024-03-29T11:20:01.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.761+0000 7ff04f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:01.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:01.761+0000 7ff04f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.100+0000 7ff04d7fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:02.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.100+0000 7ff04d7fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:02.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.100+0000 7ff04d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:02.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.100+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff068075f10 con 0x7ff068143d10 2024-03-29T11:20:02.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.101+0000 7ff04f7fe640 1 -- 172.21.15.19:0/3971006732 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff058052400 con 0x7ff068143d10 2024-03-29T11:20:02.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.101+0000 7ff04f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:02.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.101+0000 7ff04f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 10 librados: watch_flush enter 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 10 librados: watch_flush exit 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 10 client.4240.objecter successfully canceled tick 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff05406db90 legacy=0x7ff054070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 10 monclient: shutdown 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:02.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.106+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 >> v1:172.21.15.62:6789/0 conn(0x7ff068143d10 legacy=0x7ff06813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.107+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 shutdown_connections 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.107+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 >> 172.21.15.19:0/3971006732 conn(0x7ff068066a00 msgr2=0x7ff068066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.108+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 shutdown_connections 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.108+0000 7ff04d7fa640 1 -- 172.21.15.19:0/3971006732 wait complete. 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:02.109+0000 7ff04d7fa640 1 librados: shutdown 2024-03-29T11:20:02.102 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:08.910 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.084+0000 7f0a3a863640 10 monclient: get_monmap_and_config 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.084+0000 7f0a3a863640 10 monclient: build_initial_monmap 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.084+0000 7f0a3a863640 10 monclient: monmap: 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:09.085249+0000 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:09.085249+0000 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.061 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:09.062 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:09.062 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:09.062 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.086+0000 7f0a3a863640 1 Processor -- start 2024-03-29T11:20:09.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.086+0000 7f0a3a863640 1 -- start start 2024-03-29T11:20:09.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.086+0000 7f0a3a863640 10 monclient: init 2024-03-29T11:20:09.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:09.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:09.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient(hunting): picked mon.noname-c con 0x7f0a340af2e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient(hunting): picked mon.noname-b con 0x7f0a3406d8c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient(hunting): picked mon.noname-a con 0x7f0a340a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a340ac4f0 con 0x7f0a340a9750 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a340af770 con 0x7f0a340af2e0 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a340b00c0 con 0x7f0a3406d8c0 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:09.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.087+0000 7f0a3a863640 10 monclient(hunting): authenticate will time out at 929.350763s 2024-03-29T11:20:09.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.092+0000 7f0a337fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f0a3406d8c0 0x7f0a341417e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:38927/0 (socket says 172.21.15.19:38927) 2024-03-29T11:20:09.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.092+0000 7f0a337fe640 1 -- 172.21.15.19:0/1064588308 learned_addr learned my addr 172.21.15.19:0/1064588308 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:09.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.093+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956818506 0 0) 0x7f0a340b00c0 con 0x7f0a3406d8c0 2024-03-29T11:20:09.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.093+0000 7f0a32ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:09.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.093+0000 7f0a32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:09.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.093+0000 7f0a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:09.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.093+0000 7f0a32ffd640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:20:09.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.093+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a1c003520 con 0x7f0a3406d8c0 2024-03-29T11:20:09.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995637655 0 0) 0x7f0a1c003520 con 0x7f0a3406d8c0 2024-03-29T11:20:09.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 >> v1:172.21.15.62:6789/0 conn(0x7f0a340af2e0 legacy=0x7f0a3413b4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:20:09.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 >> v1:172.21.15.19:6789/0 conn(0x7f0a340a9750 legacy=0x7f0a34141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 1 monclient: found mon.noname-b 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3413bbf0 con 0x7f0a3406d8c0 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:09.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:09.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:09.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0a28001690 con 0x7f0a3406d8c0 2024-03-29T11:20:09.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:09.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a32ffd640 10 monclient: dump: 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a3a863640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:20:09.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.094+0000 7f0a3a863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a28001860 con 0x7f0a3406d8c0 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a32ffd640 1 -- 172.21.15.19:0/1064588308 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0a28001cb0 con 0x7f0a3406d8c0 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a32ffd640 10 monclient: dump: 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:09.076 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a3a863640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.095+0000 7f0a3a863640 4 set_mon_vals no callback set 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.096+0000 7f0a3a863640 10 monclient: shutdown 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.096+0000 7f0a3a863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.096+0000 7f0a3a863640 1 -- 172.21.15.19:0/1064588308 >> v1:172.21.15.19:6790/0 conn(0x7f0a3406d8c0 legacy=0x7f0a341417e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.096+0000 7f0a3a863640 1 -- 172.21.15.19:0/1064588308 shutdown_connections 2024-03-29T11:20:09.077 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.096+0000 7f0a3a863640 1 -- 172.21.15.19:0/1064588308 >> 172.21.15.19:0/1064588308 conn(0x7f0a340b3a00 msgr2=0x7f0a340b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:09.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.101+0000 7f0a3a863640 1 -- 172.21.15.19:0/1064588308 shutdown_connections 2024-03-29T11:20:09.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.101+0000 7f0a3a863640 1 -- 172.21.15.19:0/1064588308 wait complete. 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.102+0000 7f0a3a863640 10 monclient: build_initial_monmap 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.102+0000 7f0a3a863640 10 monclient: monmap: 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:09.079 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.102+0000 7f0a3a863640 1 librados: starting msgr at 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.102+0000 7f0a3a863640 1 librados: starting objecter 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.102+0000 7f0a3a863640 1 Processor -- start 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.103+0000 7f0a3a863640 1 -- start start 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.103+0000 7f0a3a863640 1 librados: setting wanted keys 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.103+0000 7f0a3a863640 1 librados: calling monclient init 2024-03-29T11:20:09.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.103+0000 7f0a3a863640 10 monclient: init 2024-03-29T11:20:09.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.104+0000 7f0a3a863640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:09.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.104+0000 7f0a3a863640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:09.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.104+0000 7f0a3a863640 10 monclient(hunting): picked mon.noname-a con 0x7f0a340a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:09.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.104+0000 7f0a3a863640 10 monclient(hunting): picked mon.noname-c con 0x7f0a3413bf80 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:09.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.108+0000 7f0a3a863640 10 monclient(hunting): picked mon.noname-b con 0x7f0a341dfda0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:09.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a3a863640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3406e600 con 0x7f0a340a9750 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a3a863640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3406f370 con 0x7f0a341dfda0 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a33fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f0a340a9750 0x7f0a3406def0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42653/0 (socket says 172.21.15.19:42653) 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a33fff640 1 -- 172.21.15.19:0/2442387787 learned_addr learned my addr 172.21.15.19:0/2442387787 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3406f650 con 0x7f0a3413bf80 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a30ff9640 10 client.?.objecter ms_handle_connect 0x7f0a340a9750 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a3a863640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.110+0000 7f0a3a863640 10 monclient(hunting): authenticate will time out at 929.373932s 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002968594 0 0) 0x7f0a3406e600 con 0x7f0a340a9750 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:09.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a20004eb0 con 0x7f0a340a9750 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 client.?.objecter ms_handle_connect 0x7f0a341dfda0 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096232026 0 0) 0x7f0a3406f370 con 0x7f0a341dfda0 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3406e600 con 0x7f0a341dfda0 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 client.?.objecter ms_handle_connect 0x7f0a3413bf80 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369642345 0 0) 0x7f0a3406f650 con 0x7f0a3413bf80 2024-03-29T11:20:09.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a3406f370 con 0x7f0a3413bf80 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4130069507 0 0) 0x7f0a20004eb0 con 0x7f0a340a9750 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a3406f650 con 0x7f0a340a9750 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0a24001870 con 0x7f0a340a9750 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.111+0000 7f0a30ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:09.094 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 792736811 0 0) 0x7f0a3406e600 con 0x7f0a341dfda0 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a20004eb0 con 0x7f0a341dfda0 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0a2c001a80 con 0x7f0a341dfda0 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:09.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359418843 0 0) 0x7f0a3406f370 con 0x7f0a3413bf80 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a3406e600 con 0x7f0a3413bf80 2024-03-29T11:20:09.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1791944970 0 0) 0x7f0a3406f650 con 0x7f0a340a9750 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 >> v1:172.21.15.19:6790/0 conn(0x7f0a3413bf80 legacy=0x7f0a3413c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 >> v1:172.21.15.62:6789/0 conn(0x7f0a341dfda0 legacy=0x7f0a341e2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 monclient: found mon.a 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3413caf0 con 0x7f0a340a9750 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 10 monclient: _renew_subs 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:09.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a3413edc0 con 0x7f0a340a9750 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a24001d80 con 0x7f0a340a9750 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0a24004a30 con 0x7f0a340a9750 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a30ff9640 10 monclient: dump: 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:09.098 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a327fc640 4 set_mon_vals no callback set 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 10 monclient: _renew_subs 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a341e3a90 con 0x7f0a340a9750 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.112+0000 7f0a3a863640 1 librados: init done 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.114+0000 7f0a3a863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.114+0000 7f0a3a863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.114+0000 7f0a3a863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:09.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.114+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a3413efb0 con 0x7f0a340a9750 2024-03-29T11:20:09.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f0a2401aa00 con 0x7f0a340a9750 2024-03-29T11:20:09.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f0a24084650 con 0x7f0a340a9750 2024-03-29T11:20:09.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a30ff9640 10 client.4263.objecter ms_dispatch 0x7f0a34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:09.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a30ff9640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:09.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a30ff9640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:09.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a337fe640 10 monclient: get_auth_request con 0x7f0a2006e180 auth_method 0 2024-03-29T11:20:09.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.118+0000 7f0a30ff9640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:20:09.101 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.119+0000 7f0a30ff9640 10 client.4263.objecter ms_handle_connect 0x7f0a2006e180 2024-03-29T11:20:09.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.129+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a2404fea0 con 0x7f0a340a9750 2024-03-29T11:20:09.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.129+0000 7f0a30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:09.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.129+0000 7f0a30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:09.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.455+0000 7f0a3a863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:09.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.455+0000 7f0a3a863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:09.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.455+0000 7f0a3a863640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:09.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.455+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a3413e500 con 0x7f0a340a9750 2024-03-29T11:20:09.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.458+0000 7f0a30ff9640 1 -- 172.21.15.19:0/2442387787 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f0a2404fea0 con 0x7f0a340a9750 2024-03-29T11:20:09.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.458+0000 7f0a30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:09.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.458+0000 7f0a30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:09.435 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:09.435 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:09.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.462+0000 7f0a3a863640 10 librados: watch_flush enter 2024-03-29T11:20:09.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.462+0000 7f0a3a863640 10 librados: watch_flush exit 2024-03-29T11:20:09.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.462+0000 7f0a3a863640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:20:09.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.462+0000 7f0a3a863640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:20:09.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.464+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 >> v1:172.21.15.19:6816/3721232097 conn(0x7f0a2006e180 legacy=0x7f0a20070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:09.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.464+0000 7f0a3a863640 10 monclient: shutdown 2024-03-29T11:20:09.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.464+0000 7f0a3a863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:09.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.464+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 >> v1:172.21.15.19:6789/0 conn(0x7f0a340a9750 legacy=0x7f0a3406def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:09.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.464+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 shutdown_connections 2024-03-29T11:20:09.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.464+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 >> 172.21.15.19:0/2442387787 conn(0x7f0a340b3a00 msgr2=0x7f0a340b1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:09.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.467+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 shutdown_connections 2024-03-29T11:20:09.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.467+0000 7f0a3a863640 1 -- 172.21.15.19:0/2442387787 wait complete. 2024-03-29T11:20:09.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:09.468+0000 7f0a3a863640 1 librados: shutdown 2024-03-29T11:20:09.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:16.269 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:16.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.440+0000 7f9642da8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:16.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.440+0000 7f9642da8640 10 monclient: build_initial_monmap 2024-03-29T11:20:16.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.440+0000 7f9642da8640 10 monclient: monmap: 2024-03-29T11:20:16.417 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:16.417 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:16.417 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:16.440951+0000 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:16.440951+0000 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:16.418 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.443+0000 7f9642da8640 1 Processor -- start 2024-03-29T11:20:16.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.443+0000 7f9642da8640 1 -- start start 2024-03-29T11:20:16.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.443+0000 7f9642da8640 10 monclient: init 2024-03-29T11:20:16.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:16.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:16.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient(hunting): picked mon.noname-a con 0x7f963c0ab170 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:16.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient(hunting): picked mon.noname-b con 0x7f963c13f4a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:16.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient(hunting): picked mon.noname-c con 0x7f963c0aeff0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:16.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c13f010 con 0x7f963c0ab170 2024-03-29T11:20:16.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0ab6c0 con 0x7f963c0aeff0 2024-03-29T11:20:16.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c0ab9a0 con 0x7f963c13f4a0 2024-03-29T11:20:16.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:16.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.444+0000 7f9642da8640 10 monclient(hunting): authenticate will time out at 936.707198s 2024-03-29T11:20:16.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.447+0000 7f963bfff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f963c13f4a0 0x7f963c13f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52277/0 (socket says 172.21.15.19:52277) 2024-03-29T11:20:16.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.447+0000 7f963bfff640 1 -- 172.21.15.19:0/285322906 learned_addr learned my addr 172.21.15.19:0/285322906 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:16.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356641653 0 0) 0x7f963c0ab9a0 con 0x7f963c13f4a0 2024-03-29T11:20:16.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:16.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:16.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9624003520 con 0x7f963c13f4a0 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930623783 0 0) 0x7f963c0ab6c0 con 0x7f963c0aeff0 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:16.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.448+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c0ab9a0 con 0x7f963c0aeff0 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441705051 0 0) 0x7f963c0ab9a0 con 0x7f963c0aeff0 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 >> v1:172.21.15.19:6790/0 conn(0x7f963c13f4a0 legacy=0x7f963c13f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 >> v1:172.21.15.19:6789/0 conn(0x7f963c0ab170 legacy=0x7f963c13c3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:20:16.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c145830 con 0x7f963c0aeff0 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f96340019f0 con 0x7f963c0aeff0 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f963b7fe640 10 monclient: dump: 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.427 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f9642da8640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.449+0000 7f9642da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.450+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9634001b50 con 0x7f963c0aeff0 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.450+0000 7f963b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.450+0000 7f963b7fe640 1 -- 172.21.15.19:0/285322906 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9634001fa0 con 0x7f963c0aeff0 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.450+0000 7f963b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.450+0000 7f963b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.450+0000 7f963b7fe640 10 monclient: dump: 2024-03-29T11:20:16.428 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:16.429 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.451+0000 7f9642da8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.451+0000 7f9642da8640 4 set_mon_vals no callback set 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.452+0000 7f9642da8640 10 monclient: shutdown 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.452+0000 7f9642da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.452+0000 7f9642da8640 1 -- 172.21.15.19:0/285322906 >> v1:172.21.15.62:6789/0 conn(0x7f963c0aeff0 legacy=0x7f963c14ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.452+0000 7f9642da8640 1 -- 172.21.15.19:0/285322906 shutdown_connections 2024-03-29T11:20:16.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.452+0000 7f9642da8640 1 -- 172.21.15.19:0/285322906 >> 172.21.15.19:0/285322906 conn(0x7f963c0b35e0 msgr2=0x7f963c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.453+0000 7f9642da8640 1 -- 172.21.15.19:0/285322906 shutdown_connections 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.453+0000 7f9642da8640 1 -- 172.21.15.19:0/285322906 wait complete. 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.454+0000 7f9642da8640 10 monclient: build_initial_monmap 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.454+0000 7f9642da8640 10 monclient: monmap: 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:16.431 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.454+0000 7f9642da8640 1 librados: starting msgr at 2024-03-29T11:20:16.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.454+0000 7f9642da8640 1 librados: starting objecter 2024-03-29T11:20:16.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.454+0000 7f9642da8640 1 Processor -- start 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.456+0000 7f9642da8640 1 -- start start 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.456+0000 7f9642da8640 1 librados: setting wanted keys 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.456+0000 7f9642da8640 1 librados: calling monclient init 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.456+0000 7f9642da8640 10 monclient: init 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient(hunting): picked mon.noname-c con 0x7f963c0ab170 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:16.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient(hunting): picked mon.noname-a con 0x7f963c0aeff0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:16.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient(hunting): picked mon.noname-b con 0x7f963c13f4a0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:16.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c142ec0 con 0x7f963c0aeff0 2024-03-29T11:20:16.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c144b50 con 0x7f963c13f4a0 2024-03-29T11:20:16.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c143b60 con 0x7f963c0ab170 2024-03-29T11:20:16.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:16.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.457+0000 7f9642da8640 10 monclient(hunting): authenticate will time out at 936.720970s 2024-03-29T11:20:16.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.461+0000 7f9641345640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f963c13f4a0 0x7f963c1e78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:36671/0 (socket says 172.21.15.19:36671) 2024-03-29T11:20:16.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.461+0000 7f9641345640 1 -- 172.21.15.19:0/1587588853 learned_addr learned my addr 172.21.15.19:0/1587588853 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c13f4a0 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790342044 0 0) 0x7f963c144b50 con 0x7f963c13f4a0 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:16.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:20:16.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9628004eb0 con 0x7f963c13f4a0 2024-03-29T11:20:16.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c0aeff0 2024-03-29T11:20:16.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 client.?.objecter ms_handle_connect 0x7f963c0ab170 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974575016 0 0) 0x7f963c143b60 con 0x7f963c0ab170 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c144b50 con 0x7f963c0ab170 2024-03-29T11:20:16.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140054208 0 0) 0x7f963c142ec0 con 0x7f963c0aeff0 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.466+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c143b60 con 0x7f963c0aeff0 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675203816 0 0) 0x7f963c144b50 con 0x7f963c0ab170 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c142ec0 con 0x7f963c0ab170 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 625355786 0 0) 0x7f963c143b60 con 0x7f963c0aeff0 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f963c144b50 con 0x7f963c0aeff0 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f962c001810 con 0x7f963c0ab170 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): dump: 2024-03-29T11:20:16.445 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9630001810 con 0x7f963c0aeff0 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:16.446 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient(hunting): dump: 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2352538093 0 0) 0x7f963c142ec0 con 0x7f963c0ab170 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 >> v1:172.21.15.62:6789/0 conn(0x7f963c13f4a0 legacy=0x7f963c1e78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:16.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 >> v1:172.21.15.19:6789/0 conn(0x7f963c0aeff0 legacy=0x7f963c1e7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:16.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:16.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 monclient: found mon.c 2024-03-29T11:20:16.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:16.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c145250 con 0x7f963c0ab170 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.467+0000 7f96397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 10 monclient: _renew_subs 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f962c001970 con 0x7f963c0ab170 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c1454a0 con 0x7f963c0ab170 2024-03-29T11:20:16.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 10 monclient: _renew_subs 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c1ef790 con 0x7f963c0ab170 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f9642da8640 1 librados: init done 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f96397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f962c003790 con 0x7f963c0ab170 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f96397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f96397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f96397fa640 10 monclient: dump: 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.451 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:16.452 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:16.452 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:16.452 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:16.452 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:16.452 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:16.452 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:16.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.468+0000 7f963a7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:16.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.470+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f962c019760 con 0x7f963c0ab170 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.471+0000 7f9642da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.471+0000 7f9642da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.471+0000 7f9642da8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.471+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c005080 con 0x7f963c0ab170 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.471+0000 7f963bfff640 10 monclient: get_auth_request con 0x7f962806e020 auth_method 0 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.472+0000 7f96397fa640 10 client.4268.objecter ms_handle_connect 0x7f962806e020 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.472+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f962c083770 con 0x7f963c0ab170 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.472+0000 7f96397fa640 10 client.4268.objecter ms_dispatch 0x7f963c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.472+0000 7f96397fa640 3 client.4268.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.472+0000 7f96397fa640 3 client.4268.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:16.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.472+0000 7f96397fa640 20 client.4268.objecter dump_active .. 0 homeless 2024-03-29T11:20:16.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.481+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f962c04f050 con 0x7f963c0ab170 2024-03-29T11:20:16.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.481+0000 7f96397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:16.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.481+0000 7f96397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:16.794 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.818+0000 7f9642da8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:16.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.818+0000 7f9642da8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:16.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.818+0000 7f9642da8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:16.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.818+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f961c005330 con 0x7f963c0ab170 2024-03-29T11:20:16.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.822+0000 7f96397fa640 1 -- 172.21.15.19:0/1587588853 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f962c056ba0 con 0x7f963c0ab170 2024-03-29T11:20:16.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.822+0000 7f96397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:16.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.822+0000 7f96397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:16.799 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:16.799 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.826+0000 7f9642da8640 10 librados: watch_flush enter 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.826+0000 7f9642da8640 10 librados: watch_flush exit 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.826+0000 7f9642da8640 20 client.4268.objecter shutdown clearing up homeless session... 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.826+0000 7f9642da8640 10 client.4268.objecter successfully canceled tick 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.826+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 >> v1:172.21.15.19:6816/3721232097 conn(0x7f962806e020 legacy=0x7f96280704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.826+0000 7f9642da8640 10 monclient: shutdown 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:16.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 >> v1:172.21.15.19:6790/0 conn(0x7f963c0ab170 legacy=0x7f963c143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 shutdown_connections 2024-03-29T11:20:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 >> 172.21.15.19:0/1587588853 conn(0x7f963c0b35e0 msgr2=0x7f963c0af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 shutdown_connections 2024-03-29T11:20:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 1 -- 172.21.15.19:0/1587588853 wait complete. 2024-03-29T11:20:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:16.827+0000 7f9642da8640 1 librados: shutdown 2024-03-29T11:20:16.822 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:23.628 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.810+0000 7f1460f64640 10 monclient: get_monmap_and_config 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.810+0000 7f1460f64640 10 monclient: build_initial_monmap 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.810+0000 7f1460f64640 10 monclient: monmap: 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:23.811164+0000 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:23.811164+0000 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:23.788 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:23.789 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.811+0000 7f1460f64640 1 Processor -- start 2024-03-29T11:20:23.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.811+0000 7f1460f64640 1 -- start start 2024-03-29T11:20:23.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.811+0000 7f1460f64640 10 monclient: init 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient(hunting): picked mon.noname-c con 0x7f1454034850 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient(hunting): picked mon.noname-a con 0x7f14540c9550 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient(hunting): picked mon.noname-b con 0x7f145402aac0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145402c440 con 0x7f14540c9550 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145402aed0 con 0x7f1454034850 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145402b190 con 0x7f145402aac0 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:23.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.812+0000 7f1460f64640 10 monclient(hunting): authenticate will time out at 944.075889s 2024-03-29T11:20:23.795 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.816+0000 7f145affd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f14540c9550 0x7f14540cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48821/0 (socket says 172.21.15.19:48821) 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.816+0000 7f145affd640 1 -- 172.21.15.19:0/594720765 learned_addr learned my addr 172.21.15.19:0/594720765 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687105402 0 0) 0x7f145402c440 con 0x7f14540c9550 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:20:23.796 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f144c003520 con 0x7f14540c9550 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262980806 0 0) 0x7f144c003520 con 0x7f14540c9550 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 >> v1:172.21.15.19:6790/0 conn(0x7f145402aac0 legacy=0x7f14540cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 >> v1:172.21.15.62:6789/0 conn(0x7f1454034850 legacy=0x7f14540c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:20:23.797 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1454033060 con 0x7f14540c9550 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f1460f64640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f1460f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1448001690 con 0x7f14540c9550 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: dump: 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.798 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f1460f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1448001920 con 0x7f14540c9550 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 1 -- 172.21.15.19:0/594720765 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1448001cb0 con 0x7f14540c9550 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f145a7fc640 10 monclient: dump: 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:23.799 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f1460f64640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:23.800 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.820+0000 7f1460f64640 4 set_mon_vals no callback set 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.821+0000 7f1460f64640 10 monclient: shutdown 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.821+0000 7f1460f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.821+0000 7f1460f64640 1 -- 172.21.15.19:0/594720765 >> v1:172.21.15.19:6789/0 conn(0x7f14540c9550 legacy=0x7f14540cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.821+0000 7f1460f64640 1 -- 172.21.15.19:0/594720765 shutdown_connections 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.821+0000 7f1460f64640 1 -- 172.21.15.19:0/594720765 >> 172.21.15.19:0/594720765 conn(0x7f14540058f0 msgr2=0x7f1454005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.824+0000 7f1460f64640 1 -- 172.21.15.19:0/594720765 shutdown_connections 2024-03-29T11:20:23.801 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.824+0000 7f1460f64640 1 -- 172.21.15.19:0/594720765 wait complete. 2024-03-29T11:20:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.825+0000 7f1460f64640 10 monclient: build_initial_monmap 2024-03-29T11:20:23.803 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.825+0000 7f1460f64640 10 monclient: monmap: 2024-03-29T11:20:23.803 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:23.803 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 1 librados: starting msgr at 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 1 librados: starting objecter 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 1 Processor -- start 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 1 -- start start 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 1 librados: setting wanted keys 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 1 librados: calling monclient init 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.826+0000 7f1460f64640 10 monclient: init 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.827+0000 7f1460f64640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:23.805 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.827+0000 7f1460f64640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:23.807 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.831+0000 7f1460f64640 10 monclient(hunting): picked mon.noname-a con 0x7f145402aac0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 10 monclient(hunting): picked mon.noname-c con 0x7f1454034850 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 10 monclient(hunting): picked mon.noname-b con 0x7f14540c9550 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14540b1a30 con 0x7f145402aac0 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14540bef90 con 0x7f14540c9550 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14540bd000 con 0x7f1454034850 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:23.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.833+0000 7f1460f64640 10 monclient(hunting): authenticate will time out at 944.096509s 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f145affd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1454034850 0x7f14540b3d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:37367/0 (socket says 172.21.15.19:37367) 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f145affd640 1 -- 172.21.15.19:0/160476770 learned_addr learned my addr 172.21.15.19:0/160476770 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f1447fff640 10 client.?.objecter ms_handle_connect 0x7f14540c9550 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f1447fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f1447fff640 10 client.?.objecter ms_handle_connect 0x7f145402aac0 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f1447fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f1447fff640 10 client.?.objecter ms_handle_connect 0x7f1454034850 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.837+0000 7f1447fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:23.815 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499938548 0 0) 0x7f14540bd000 con 0x7f1454034850 2024-03-29T11:20:23.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:23.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:23.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:23.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:20:23.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1440004eb0 con 0x7f1454034850 2024-03-29T11:20:23.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541881110 0 0) 0x7f14540bef90 con 0x7f14540c9550 2024-03-29T11:20:23.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14540bd000 con 0x7f14540c9550 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135379842 0 0) 0x7f1440004eb0 con 0x7f1454034850 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14540bef90 con 0x7f1454034850 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1448001610 con 0x7f1454034850 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): dump: 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:23.826 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2187453101 0 0) 0x7f14540bd000 con 0x7f14540c9550 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1440004eb0 con 0x7f14540c9550 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f145c04fdd0 con 0x7f14540c9550 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:23.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.838+0000 7f1447fff640 10 monclient(hunting): dump: 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1714712435 0 0) 0x7f14540bef90 con 0x7f1454034850 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 >> v1:172.21.15.62:6789/0 conn(0x7f14540c9550 legacy=0x7f14540b8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 >> v1:172.21.15.19:6789/0 conn(0x7f145402aac0 legacy=0x7f14540bc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:23.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 monclient: found mon.c 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14540b4470 con 0x7f1454034850 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 10 monclient: _renew_subs 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14540b8f10 con 0x7f1454034850 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 10 monclient: _renew_subs 2024-03-29T11:20:23.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14540be910 con 0x7f1454034850 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1460f64640 1 librados: init done 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1448004a90 con 0x7f1454034850 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1448004da0 con 0x7f1454034850 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.839+0000 7f1447fff640 10 monclient: dump: 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:23.830 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.840+0000 7f14597fa640 4 set_mon_vals no callback set 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.841+0000 7f1460f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1460f64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1460f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f144801ad70 con 0x7f1454034850 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143c005080 con 0x7f1454034850 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f1448084b70 con 0x7f1454034850 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1447fff640 10 client.4271.objecter ms_dispatch 0x7f145c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:23.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1447fff640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1447fff640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.842+0000 7f1447fff640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.843+0000 7f145b7fe640 10 monclient: get_auth_request con 0x7f144006ddb0 auth_method 0 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.843+0000 7f1447fff640 10 client.4271.objecter ms_handle_connect 0x7f144006ddb0 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.850+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1448084e60 con 0x7f1454034850 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.850+0000 7f1447fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:23.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:23.850+0000 7f1447fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:24.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.182+0000 7f1460f64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:24.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.182+0000 7f1460f64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:24.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.183+0000 7f1460f64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:24.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.183+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f143c005330 con 0x7f1454034850 2024-03-29T11:20:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.183+0000 7f1447fff640 1 -- 172.21.15.19:0/160476770 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f1448050450 con 0x7f1454034850 2024-03-29T11:20:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.184+0000 7f1447fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.184+0000 7f1447fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:24.161 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:24.161 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 10 librados: watch_flush enter 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 10 librados: watch_flush exit 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 10 client.4271.objecter successfully canceled tick 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 >> v1:172.21.15.19:6816/3721232097 conn(0x7f144006ddb0 legacy=0x7f1440070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 10 monclient: shutdown 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 >> v1:172.21.15.19:6790/0 conn(0x7f1454034850 legacy=0x7f14540b3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 shutdown_connections 2024-03-29T11:20:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.188+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 >> 172.21.15.19:0/160476770 conn(0x7f14540058f0 msgr2=0x7f14540043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:24.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.189+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 shutdown_connections 2024-03-29T11:20:24.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.189+0000 7f1460f64640 1 -- 172.21.15.19:0/160476770 wait complete. 2024-03-29T11:20:24.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:24.189+0000 7f1460f64640 1 librados: shutdown 2024-03-29T11:20:24.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:30.988 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:31.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.166+0000 7fde73196640 10 monclient: get_monmap_and_config 2024-03-29T11:20:31.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.166+0000 7fde73196640 10 monclient: build_initial_monmap 2024-03-29T11:20:31.142 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.166+0000 7fde73196640 10 monclient: monmap: 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:31.166867+0000 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:31.166867+0000 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:31.143 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.167+0000 7fde73196640 1 Processor -- start 2024-03-29T11:20:31.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.167+0000 7fde73196640 1 -- start start 2024-03-29T11:20:31.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.167+0000 7fde73196640 10 monclient: init 2024-03-29T11:20:31.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.167+0000 7fde73196640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:31.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.167+0000 7fde73196640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:31.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 10 monclient(hunting): picked mon.noname-a con 0x7fde6c06d9d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:31.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 10 monclient(hunting): picked mon.noname-b con 0x7fde6c13e820 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:31.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 10 monclient(hunting): picked mon.noname-c con 0x7fde6c0af1d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:31.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0aed10 con 0x7fde6c06d9d0 2024-03-29T11:20:31.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0afd50 con 0x7fde6c0af1d0 2024-03-29T11:20:31.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c0ab490 con 0x7fde6c13e820 2024-03-29T11:20:31.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:31.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.168+0000 7fde73196640 10 monclient(hunting): authenticate will time out at 951.431069s 2024-03-29T11:20:31.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.175+0000 7fde6bfff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fde6c13e820 0x7fde6c13ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54013/0 (socket says 172.21.15.19:54013) 2024-03-29T11:20:31.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.175+0000 7fde6bfff640 1 -- 172.21.15.19:0/2511916203 learned_addr learned my addr 172.21.15.19:0/2511916203 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:31.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668248811 0 0) 0x7fde6c0ab490 con 0x7fde6c13e820 2024-03-29T11:20:31.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:31.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:31.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:31.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:20:31.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54003520 con 0x7fde6c13e820 2024-03-29T11:20:31.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3622552154 0 0) 0x7fde54003520 con 0x7fde6c13e820 2024-03-29T11:20:31.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.176+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 >> v1:172.21.15.62:6789/0 conn(0x7fde6c0af1d0 legacy=0x7fde6c0af640 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 >> v1:172.21.15.19:6789/0 conn(0x7fde6c06d9d0 legacy=0x7fde6c0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c0ab770 con 0x7fde6c13e820 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fde64001a10 con 0x7fde6c13e820 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: dump: 2024-03-29T11:20:31.155 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde73196640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde73196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde64001be0 con 0x7fde6c13e820 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:31.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde73196640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 1 -- 172.21.15.19:0/2511916203 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fde64002030 con 0x7fde6c13e820 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde73196640 4 set_mon_vals no callback set 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.177+0000 7fde6b7fe640 10 monclient: dump: 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:31.157 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:31.158 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:31.158 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.178+0000 7fde73196640 10 monclient: shutdown 2024-03-29T11:20:31.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.178+0000 7fde73196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:31.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.178+0000 7fde73196640 1 -- 172.21.15.19:0/2511916203 >> v1:172.21.15.19:6790/0 conn(0x7fde6c13e820 legacy=0x7fde6c13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.179+0000 7fde73196640 1 -- 172.21.15.19:0/2511916203 shutdown_connections 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.179+0000 7fde73196640 1 -- 172.21.15.19:0/2511916203 >> 172.21.15.19:0/2511916203 conn(0x7fde6c06b660 msgr2=0x7fde6c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.179+0000 7fde73196640 1 -- 172.21.15.19:0/2511916203 shutdown_connections 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.179+0000 7fde73196640 1 -- 172.21.15.19:0/2511916203 wait complete. 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 10 monclient: build_initial_monmap 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 10 monclient: monmap: 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.160 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 1 librados: starting msgr at 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 1 librados: starting objecter 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 1 Processor -- start 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 1 -- start start 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 1 librados: setting wanted keys 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 1 librados: calling monclient init 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.180+0000 7fde73196640 10 monclient: init 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient(hunting): picked mon.noname-c con 0x7fde6c06d9d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:31.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient(hunting): picked mon.noname-b con 0x7fde6c0af1d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:31.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient(hunting): picked mon.noname-a con 0x7fde6c13e820 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:31.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c1e4db0 con 0x7fde6c13e820 2024-03-29T11:20:31.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c06e740 con 0x7fde6c0af1d0 2024-03-29T11:20:31.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6c06ea20 con 0x7fde6c06d9d0 2024-03-29T11:20:31.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:31.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.181+0000 7fde73196640 10 monclient(hunting): authenticate will time out at 951.444842s 2024-03-29T11:20:31.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.188+0000 7fde71733640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fde6c13e820 0x7fde6c1e46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:55685/0 (socket says 172.21.15.19:55685) 2024-03-29T11:20:31.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.188+0000 7fde71733640 1 -- 172.21.15.19:0/3399640454 learned_addr learned my addr 172.21.15.19:0/3399640454 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:31.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.189+0000 7fde697fa640 10 client.?.objecter ms_handle_connect 0x7fde6c13e820 2024-03-29T11:20:31.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.189+0000 7fde697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:31.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.191+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824319903 0 0) 0x7fde6c1e4db0 con 0x7fde6c13e820 2024-03-29T11:20:31.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.191+0000 7fde697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:31.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.191+0000 7fde697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:31.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.191+0000 7fde697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:31.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.191+0000 7fde697fa640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.191+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde58004eb0 con 0x7fde6c13e820 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 client.?.objecter ms_handle_connect 0x7fde6c06d9d0 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 client.?.objecter ms_handle_connect 0x7fde6c0af1d0 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510060747 0 0) 0x7fde6c06ea20 con 0x7fde6c06d9d0 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde6c1e4db0 con 0x7fde6c06d9d0 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3762930510 0 0) 0x7fde58004eb0 con 0x7fde6c13e820 2024-03-29T11:20:31.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde6c06ea20 con 0x7fde6c13e820 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fde600017f0 con 0x7fde6c13e820 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): dump: 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:31.175 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2391735601 0 0) 0x7fde6c1e4db0 con 0x7fde6c06d9d0 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde58004eb0 con 0x7fde6c06d9d0 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fde5c0017d0 con 0x7fde6c06d9d0 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): dump: 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.176 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352595905 0 0) 0x7fde6c06e740 con 0x7fde6c0af1d0 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde6c1e4db0 con 0x7fde6c0af1d0 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549644705 0 0) 0x7fde6c06ea20 con 0x7fde6c13e820 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.192+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 >> v1:172.21.15.19:6790/0 conn(0x7fde6c06d9d0 legacy=0x7fde6c0b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 >> v1:172.21.15.62:6789/0 conn(0x7fde6c0af1d0 legacy=0x7fde6c1e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:31.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 1 monclient: found mon.a 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6c06ed00 con 0x7fde6c13e820 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 10 monclient: _renew_subs 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6c0b3d10 con 0x7fde6c13e820 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 10 monclient: _renew_subs 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:31.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde6c0b3880 con 0x7fde6c13e820 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.193+0000 7fde73196640 1 librados: init done 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde60001d00 con 0x7fde6c13e820 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde73196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fde60004aa0 con 0x7fde6c13e820 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde73196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde73196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6c0b3f50 con 0x7fde6c13e820 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.194+0000 7fde6a7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.195+0000 7fde697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.195+0000 7fde697fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.195+0000 7fde697fa640 10 monclient: dump: 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:31.179 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.196+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fde6001aa70 con 0x7fde6c13e820 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.196+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fde60084a20 con 0x7fde6c13e820 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.197+0000 7fde697fa640 10 client.4278.objecter ms_dispatch 0x7fde6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.197+0000 7fde697fa640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.197+0000 7fde697fa640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.197+0000 7fde697fa640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:20:31.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.197+0000 7fde70f32640 10 monclient: get_auth_request con 0x7fde5806df90 auth_method 0 2024-03-29T11:20:31.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.200+0000 7fde697fa640 10 client.4278.objecter ms_handle_connect 0x7fde5806df90 2024-03-29T11:20:31.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.205+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde6c0b3f50 con 0x7fde6c13e820 2024-03-29T11:20:31.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.205+0000 7fde697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:31.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.205+0000 7fde697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:31.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.532+0000 7fde73196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:31.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.532+0000 7fde73196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:31.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.532+0000 7fde73196640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:20:31.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.532+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde6c0b3f50 con 0x7fde6c13e820 2024-03-29T11:20:31.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.533+0000 7fde697fa640 1 -- 172.21.15.19:0/3399640454 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fde6c0b3f50 con 0x7fde6c13e820 2024-03-29T11:20:31.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.533+0000 7fde697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:31.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.533+0000 7fde697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:31.510 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:31.510 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.536+0000 7fde73196640 10 librados: watch_flush enter 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.536+0000 7fde73196640 10 librados: watch_flush exit 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 >> v1:172.21.15.19:6816/3721232097 conn(0x7fde5806df90 legacy=0x7fde58070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 10 monclient: shutdown 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 >> v1:172.21.15.19:6789/0 conn(0x7fde6c13e820 legacy=0x7fde6c1e46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 shutdown_connections 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 >> 172.21.15.19:0/3399640454 conn(0x7fde6c06b660 msgr2=0x7fde6c14d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 shutdown_connections 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.537+0000 7fde73196640 1 -- 172.21.15.19:0/3399640454 wait complete. 2024-03-29T11:20:31.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:31.538+0000 7fde73196640 1 librados: shutdown 2024-03-29T11:20:31.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:38.338 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.509+0000 7fe4499f6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.509+0000 7fe4499f6640 10 monclient: build_initial_monmap 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.509+0000 7fe4499f6640 10 monclient: monmap: 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:38.510430+0000 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:38.510430+0000 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:38.486 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.510+0000 7fe4499f6640 1 Processor -- start 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.510+0000 7fe4499f6640 1 -- start start 2024-03-29T11:20:38.487 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.510+0000 7fe4499f6640 10 monclient: init 2024-03-29T11:20:38.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:38.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:38.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe43c006e60 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:38.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe43c035e30 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:38.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe43c0c96a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:38.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0359a0 con 0x7fe43c0c96a0 2024-03-29T11:20:38.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0362b0 con 0x7fe43c006e60 2024-03-29T11:20:38.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c035670 con 0x7fe43c035e30 2024-03-29T11:20:38.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:38.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.511+0000 7fe4499f6640 10 monclient(hunting): authenticate will time out at 958.774842s 2024-03-29T11:20:38.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe442ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe43c006e60 0x7fe43c0cbcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:38762/0 (socket says 172.21.15.19:38762) 2024-03-29T11:20:38.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe442ffd640 1 -- 172.21.15.19:0/2172368719 learned_addr learned my addr 172.21.15.19:0/2172368719 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355311892 0 0) 0x7fe43c0359a0 con 0x7fe43c0c96a0 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c003520 con 0x7fe43c0c96a0 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330976816 0 0) 0x7fe43c0362b0 con 0x7fe43c006e60 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:38.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.512+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe43c0359a0 con 0x7fe43c006e60 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2439688722 0 0) 0x7fe42c003520 con 0x7fe43c0c96a0 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 >> v1:172.21.15.19:6790/0 conn(0x7fe43c035e30 legacy=0x7fe43c0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 >> v1:172.21.15.62:6789/0 conn(0x7fe43c006e60 legacy=0x7fe43c0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 1 monclient: found mon.noname-a 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c034e50 con 0x7fe43c0c96a0 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:38.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe44404f730 con 0x7fe43c0c96a0 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe441ffb640 10 monclient: dump: 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:38.492 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe4499f6640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.513+0000 7fe4499f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4440008d0 con 0x7fe43c0c96a0 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe441ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe441ffb640 1 -- 172.21.15.19:0/2172368719 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe4440758b0 con 0x7fe43c0c96a0 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe441ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe441ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe441ffb640 10 monclient: dump: 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.493 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.494 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:38.494 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:38.494 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:38.494 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:38.494 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:38.494 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe4499f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.514+0000 7fe4499f6640 4 set_mon_vals no callback set 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 10 monclient: shutdown 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 1 -- 172.21.15.19:0/2172368719 >> v1:172.21.15.19:6789/0 conn(0x7fe43c0c96a0 legacy=0x7fe43c0d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 1 -- 172.21.15.19:0/2172368719 shutdown_connections 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 1 -- 172.21.15.19:0/2172368719 >> 172.21.15.19:0/2172368719 conn(0x7fe43c005c30 msgr2=0x7fe43c006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 1 -- 172.21.15.19:0/2172368719 shutdown_connections 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.515+0000 7fe4499f6640 1 -- 172.21.15.19:0/2172368719 wait complete. 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.516+0000 7fe4499f6640 10 monclient: build_initial_monmap 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.516+0000 7fe4499f6640 10 monclient: monmap: 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:38.496 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 1 librados: starting msgr at 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 1 librados: starting objecter 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 1 Processor -- start 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 1 -- start start 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 1 librados: setting wanted keys 2024-03-29T11:20:38.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 1 librados: calling monclient init 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.517+0000 7fe4499f6640 10 monclient: init 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.518+0000 7fe4499f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.518+0000 7fe4499f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.518+0000 7fe4499f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe43c006e60 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.518+0000 7fe4499f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe43c035e30 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4499f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe43c0c96a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4499f6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c033390 con 0x7fe43c006e60 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4427fc640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe43c035e30 0x7fe43c02f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:39036/0 (socket says 172.21.15.19:39036) 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4427fc640 1 -- 172.21.15.19:0/3343995706 learned_addr learned my addr 172.21.15.19:0/3343995706 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c0cd800 con 0x7fe43c035e30 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43c02f040 con 0x7fe43c0c96a0 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4499f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe4499f6640 10 monclient(hunting): authenticate will time out at 958.782636s 2024-03-29T11:20:38.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe42bfff640 10 client.?.objecter ms_handle_connect 0x7fe43c035e30 2024-03-29T11:20:38.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.519+0000 7fe42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:38.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 client.?.objecter ms_handle_connect 0x7fe43c006e60 2024-03-29T11:20:38.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:38.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 client.?.objecter ms_handle_connect 0x7fe43c0c96a0 2024-03-29T11:20:38.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:38.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191453843 0 0) 0x7fe43c0cd800 con 0x7fe43c035e30 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe430004eb0 con 0x7fe43c035e30 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961053542 0 0) 0x7fe43c02f040 con 0x7fe43c0c96a0 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.520+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe43c0cd800 con 0x7fe43c0c96a0 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4133001241 0 0) 0x7fe430004eb0 con 0x7fe43c035e30 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe43c02f040 con 0x7fe43c035e30 2024-03-29T11:20:38.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe4380019d0 con 0x7fe43c035e30 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient(hunting): dump: 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:38.503 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2237300058 0 0) 0x7fe43c02f040 con 0x7fe43c035e30 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 >> v1:172.21.15.19:6790/0 conn(0x7fe43c0c96a0 legacy=0x7fe43c16bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 >> v1:172.21.15.19:6789/0 conn(0x7fe43c006e60 legacy=0x7fe43c032c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 monclient: found mon.b 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43c02fc70 con 0x7fe43c035e30 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:38.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe42bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 10 monclient: _renew_subs 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43c02d180 con 0x7fe43c035e30 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 10 monclient: _renew_subs 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c02ea10 con 0x7fe43c035e30 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.521+0000 7fe4499f6640 1 librados: init done 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe438001b30 con 0x7fe43c035e30 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe42bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe438003a60 con 0x7fe43c035e30 2024-03-29T11:20:38.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe42bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe42bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe42bfff640 10 monclient: dump: 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.522+0000 7fe4417fa640 4 set_mon_vals no callback set 2024-03-29T11:20:38.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.523+0000 7fe4499f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:38.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.523+0000 7fe4499f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:38.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.523+0000 7fe4499f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:38.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.524+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe424005080 con 0x7fe43c035e30 2024-03-29T11:20:38.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.526+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fe438003bc0 con 0x7fe43c035e30 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.529+0000 7fe442ffd640 10 monclient: get_auth_request con 0x7fe43006dd00 auth_method 0 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.531+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fe438084ad0 con 0x7fe43c035e30 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.531+0000 7fe42bfff640 10 client.4264.objecter ms_dispatch 0x7fe444000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.531+0000 7fe42bfff640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.531+0000 7fe42bfff640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.531+0000 7fe42bfff640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:20:38.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.531+0000 7fe42bfff640 10 client.4264.objecter ms_handle_connect 0x7fe43006dd00 2024-03-29T11:20:38.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.534+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe438086e60 con 0x7fe43c035e30 2024-03-29T11:20:38.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.534+0000 7fe42bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:38.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.534+0000 7fe42bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.904+0000 7fe4499f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.904+0000 7fe4499f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.904+0000 7fe4499f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.904+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe424005330 con 0x7fe43c035e30 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.905+0000 7fe42bfff640 1 -- 172.21.15.19:0/3343995706 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fe4380501e0 con 0x7fe43c035e30 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.905+0000 7fe42bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:38.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.905+0000 7fe42bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:38.882 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:38.883 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.909+0000 7fe4499f6640 10 librados: watch_flush enter 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.909+0000 7fe4499f6640 10 librados: watch_flush exit 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.909+0000 7fe4499f6640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.909+0000 7fe4499f6640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.909+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 >> v1:172.21.15.19:6816/3721232097 conn(0x7fe43006dd00 legacy=0x7fe4300701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.909+0000 7fe4499f6640 10 monclient: shutdown 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.910+0000 7fe4499f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:38.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.910+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 >> v1:172.21.15.62:6789/0 conn(0x7fe43c035e30 legacy=0x7fe43c02f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:38.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.910+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 shutdown_connections 2024-03-29T11:20:38.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.910+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 >> 172.21.15.19:0/3343995706 conn(0x7fe43c005c30 msgr2=0x7fe43c0c4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:38.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.910+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 shutdown_connections 2024-03-29T11:20:38.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.910+0000 7fe4499f6640 1 -- 172.21.15.19:0/3343995706 wait complete. 2024-03-29T11:20:38.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:38.911+0000 7fe4499f6640 1 librados: shutdown 2024-03-29T11:20:38.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:45.709 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.867+0000 7ff7e9fac640 10 monclient: get_monmap_and_config 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.867+0000 7ff7e9fac640 10 monclient: build_initial_monmap 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.867+0000 7ff7e9fac640 10 monclient: monmap: 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:45.868364+0000 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:45.868364+0000 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:45.844 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:45.845 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:45.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.868+0000 7ff7e9fac640 1 Processor -- start 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 1 -- start start 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient: init 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient(hunting): picked mon.noname-a con 0x7ff7e41457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient(hunting): picked mon.noname-b con 0x7ff7e413f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient(hunting): picked mon.noname-c con 0x7ff7e40a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e40aaef0 con 0x7ff7e41457f0 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e4145bd0 con 0x7ff7e40a9d70 2024-03-29T11:20:45.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e4145eb0 con 0x7ff7e413f650 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.869+0000 7ff7e9fac640 10 monclient(hunting): authenticate will time out at 966.132888s 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e2ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff7e413f650 0x7ff7e40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51134/0 (socket says 172.21.15.19:51134) 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e2ffd640 1 -- 172.21.15.19:0/4077604649 learned_addr learned my addr 172.21.15.19:0/4077604649 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950438854 0 0) 0x7ff7e4145eb0 con 0x7ff7e413f650 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7cc003520 con 0x7ff7e413f650 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3983791473 0 0) 0x7ff7cc003520 con 0x7ff7e413f650 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 >> v1:172.21.15.62:6789/0 conn(0x7ff7e40a9d70 legacy=0x7ff7e414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:45.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.870+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 >> v1:172.21.15.19:6789/0 conn(0x7ff7e41457f0 legacy=0x7ff7e413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 1 monclient: found mon.noname-b 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e4146340 con 0x7ff7e413f650 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.872+0000 7ff7e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:45.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff7dc001a10 con 0x7ff7e413f650 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e9fac640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e9fac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: dump: 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:45.853 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7dc001ca0 con 0x7ff7e413f650 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e9fac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e9fac640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e9fac640 4 set_mon_vals no callback set 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 1 -- 172.21.15.19:0/4077604649 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff7dc002030 con 0x7ff7e413f650 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.873+0000 7ff7e27fc640 10 monclient: dump: 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.854 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 10 monclient: shutdown 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 1 -- 172.21.15.19:0/4077604649 >> v1:172.21.15.19:6790/0 conn(0x7ff7e413f650 legacy=0x7ff7e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 1 -- 172.21.15.19:0/4077604649 shutdown_connections 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 1 -- 172.21.15.19:0/4077604649 >> 172.21.15.19:0/4077604649 conn(0x7ff7e4066a00 msgr2=0x7ff7e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 1 -- 172.21.15.19:0/4077604649 shutdown_connections 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.874+0000 7ff7e9fac640 1 -- 172.21.15.19:0/4077604649 wait complete. 2024-03-29T11:20:45.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.875+0000 7ff7e9fac640 10 monclient: build_initial_monmap 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.875+0000 7ff7e9fac640 10 monclient: monmap: 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.875+0000 7ff7e9fac640 1 librados: starting msgr at 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.875+0000 7ff7e9fac640 1 librados: starting objecter 2024-03-29T11:20:45.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.875+0000 7ff7e9fac640 1 Processor -- start 2024-03-29T11:20:45.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.876+0000 7ff7e9fac640 1 -- start start 2024-03-29T11:20:45.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.876+0000 7ff7e9fac640 1 librados: setting wanted keys 2024-03-29T11:20:45.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.876+0000 7ff7e9fac640 1 librados: calling monclient init 2024-03-29T11:20:45.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.876+0000 7ff7e9fac640 10 monclient: init 2024-03-29T11:20:45.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient(hunting): picked mon.noname-b con 0x7ff7e40a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient(hunting): picked mon.noname-a con 0x7ff7e413f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient(hunting): picked mon.noname-c con 0x7ff7e41457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e413a420 con 0x7ff7e413f650 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e41ce830 con 0x7ff7e40a9d70 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e41ceb10 con 0x7ff7e41457f0 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e9fac640 10 monclient(hunting): authenticate will time out at 966.140199s 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e3fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff7e41457f0 0x7ff7e4139d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51578/0 (socket says 172.21.15.19:51578) 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7e3fff640 1 -- 172.21.15.19:0/1126025555 learned_addr learned my addr 172.21.15.19:0/1126025555 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 10 client.?.objecter ms_handle_connect 0x7ff7e41457f0 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:45.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934574761 0 0) 0x7ff7e41ceb10 con 0x7ff7e41457f0 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.877+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d0004eb0 con 0x7ff7e41457f0 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092187009 0 0) 0x7ff7d0004eb0 con 0x7ff7e41457f0 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7e41ceb10 con 0x7ff7e41457f0 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff7d40018e0 con 0x7ff7e41457f0 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 10 monclient(hunting): dump: 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:45.864 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2981627132 0 0) 0x7ff7e41ceb10 con 0x7ff7e41457f0 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 >> v1:172.21.15.62:6789/0 conn(0x7ff7e40a9d70 legacy=0x7ff7e41cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.878+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 >> v1:172.21.15.19:6789/0 conn(0x7ff7e413f650 legacy=0x7ff7e4139600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 1 monclient: found mon.c 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:45.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e41cd230 con 0x7ff7e41457f0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 10 monclient: _renew_subs 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e41cba50 con 0x7ff7e41457f0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 10 monclient: _renew_subs 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e413a840 con 0x7ff7e41457f0 2024-03-29T11:20:45.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.879+0000 7ff7e9fac640 1 librados: init done 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7d40035f0 con 0x7ff7e41457f0 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff7d4003a00 con 0x7ff7e41457f0 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7cbfff640 10 monclient: dump: 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:45.867 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7e1ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.880+0000 7ff7e9fac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.881+0000 7ff7e9fac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.881+0000 7ff7e9fac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.881+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7e4075f10 con 0x7ff7e41457f0 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.883+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff7d4003b60 con 0x7ff7e41457f0 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.884+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff7d4084a60 con 0x7ff7e41457f0 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.884+0000 7ff7cbfff640 10 client.4289.objecter ms_dispatch 0x7ff7e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.884+0000 7ff7cbfff640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.884+0000 7ff7cbfff640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.884+0000 7ff7cbfff640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:20:45.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.884+0000 7ff7e37fe640 10 monclient: get_auth_request con 0x7ff7d006dc90 auth_method 0 2024-03-29T11:20:45.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.887+0000 7ff7cbfff640 10 client.4289.objecter ms_handle_connect 0x7ff7d006dc90 2024-03-29T11:20:45.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.890+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7d4050340 con 0x7ff7e41457f0 2024-03-29T11:20:45.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.890+0000 7ff7cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:45.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:45.890+0000 7ff7cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:46.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.207+0000 7ff7e9fac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:46.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.207+0000 7ff7e9fac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:46.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.207+0000 7ff7e9fac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:20:46.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.207+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7e40b3680 con 0x7ff7e41457f0 2024-03-29T11:20:46.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.208+0000 7ff7cbfff640 1 -- 172.21.15.19:0/1126025555 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff7d4052270 con 0x7ff7e41457f0 2024-03-29T11:20:46.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.208+0000 7ff7cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:46.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.208+0000 7ff7cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:46.185 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:46.185 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 10 librados: watch_flush enter 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 10 librados: watch_flush exit 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff7d006dc90 legacy=0x7ff7d0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 10 monclient: shutdown 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.210+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 >> v1:172.21.15.19:6790/0 conn(0x7ff7e41457f0 legacy=0x7ff7e4139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.211+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 shutdown_connections 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.211+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 >> 172.21.15.19:0/1126025555 conn(0x7ff7e4066a00 msgr2=0x7ff7e40b6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.211+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 shutdown_connections 2024-03-29T11:20:46.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.211+0000 7ff7e9fac640 1 -- 172.21.15.19:0/1126025555 wait complete. 2024-03-29T11:20:46.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:46.212+0000 7ff7e9fac640 1 librados: shutdown 2024-03-29T11:20:46.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:53.010 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.184+0000 7f00eb59e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.184+0000 7f00eb59e640 10 monclient: build_initial_monmap 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.184+0000 7f00eb59e640 10 monclient: monmap: 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:20:53.185250+0000 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:20:53.185250+0000 2024-03-29T11:20:53.161 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:53.162 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:53.162 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:53.162 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:20:53.162 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:20:53.162 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:53.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.185+0000 7f00eb59e640 1 Processor -- start 2024-03-29T11:20:53.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.185+0000 7f00eb59e640 1 -- start start 2024-03-29T11:20:53.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.185+0000 7f00eb59e640 10 monclient: init 2024-03-29T11:20:53.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.186+0000 7f00eb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.186+0000 7f00eb59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.186+0000 7f00eb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f00e40389e0 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f00e4038dc0 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f00e4030660 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e40c54c0 con 0x7f00e4038dc0 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e4036430 con 0x7f00e4030660 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e4039840 con 0x7f00e40389e0 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:53.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.187+0000 7f00eb59e640 10 monclient(hunting): authenticate will time out at 973.450084s 2024-03-29T11:20:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.188+0000 7f00ead9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f00e4030660 0x7f00e40d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:42228/0 (socket says 172.21.15.19:42228) 2024-03-29T11:20:53.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.188+0000 7f00ead9d640 1 -- 172.21.15.19:0/1425247388 learned_addr learned my addr 172.21.15.19:0/1425247388 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208697558 0 0) 0x7f00e4036430 con 0x7f00e4030660 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00d4003520 con 0x7f00e4030660 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603779507 0 0) 0x7f00d4003520 con 0x7f00e4030660 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 >> v1:172.21.15.19:6790/0 conn(0x7f00e40389e0 legacy=0x7f00e4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:53.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.189+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 >> v1:172.21.15.19:6789/0 conn(0x7f00e4038dc0 legacy=0x7f00e40cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 1 monclient: found mon.noname-c 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e4035690 con 0x7f00e4030660 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: _finish_auth 0 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:53.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f00ec04f730 con 0x7f00e4030660 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: dump: 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:53.169 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00eb59e640 5 monclient: authenticate success, global_id 4267 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00eb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00ec075f10 con 0x7f00e4030660 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 1 -- 172.21.15.19:0/1425247388 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f00ec0758b0 con 0x7f00e4030660 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00e959a640 10 monclient: dump: 2024-03-29T11:20:53.170 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00eb59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:53.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.190+0000 7f00eb59e640 4 set_mon_vals no callback set 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.191+0000 7f00eb59e640 10 monclient: shutdown 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.191+0000 7f00eb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.191+0000 7f00eb59e640 1 -- 172.21.15.19:0/1425247388 >> v1:172.21.15.62:6789/0 conn(0x7f00e4030660 legacy=0x7f00e40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.191+0000 7f00eb59e640 1 -- 172.21.15.19:0/1425247388 shutdown_connections 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.191+0000 7f00eb59e640 1 -- 172.21.15.19:0/1425247388 >> 172.21.15.19:0/1425247388 conn(0x7f00e4005c00 msgr2=0x7f00e4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.192+0000 7f00eb59e640 1 -- 172.21.15.19:0/1425247388 shutdown_connections 2024-03-29T11:20:53.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.192+0000 7f00eb59e640 1 -- 172.21.15.19:0/1425247388 wait complete. 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 10 monclient: build_initial_monmap 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 10 monclient: monmap: 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:20:53.178 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 1 librados: starting msgr at 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 1 librados: starting objecter 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 1 Processor -- start 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 1 -- start start 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 1 librados: setting wanted keys 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 1 librados: calling monclient init 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.193+0000 7f00eb59e640 10 monclient: init 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f00e4030660 addr v1:172.21.15.19:6790/0 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f00e40389e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:20:53.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f00e40c8240 addr v1:172.21.15.19:6789/0 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e415a560 con 0x7f00e40c8240 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e40c8670 con 0x7f00e40389e0 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e402ed70 con 0x7f00e4030660 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.194+0000 7f00eb59e640 10 monclient(hunting): authenticate will time out at 973.457444s 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.195+0000 7f00e9d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f00e40389e0 0x7f00e41604d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:42556/0 (socket says 172.21.15.19:42556) 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.195+0000 7f00e9d9b640 1 -- 172.21.15.19:0/3733969249 learned_addr learned my addr 172.21.15.19:0/3733969249 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 10 client.?.objecter ms_handle_connect 0x7f00e40389e0 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226376359 0 0) 0x7f00e0001010 con 0x7f00e40389e0 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:53.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.196+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00cc004eb0 con 0x7f00e40389e0 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.197+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663854191 0 0) 0x7f00cc004eb0 con 0x7f00e40389e0 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00e0001010 con 0x7f00e40389e0 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 client.?.objecter ms_handle_connect 0x7f00e40c8240 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f00e00017c0 con 0x7f00e40389e0 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): dump: 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:53.181 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346918069 0 0) 0x7f00e4035690 con 0x7f00e40c8240 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:53.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00cc004eb0 con 0x7f00e40c8240 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1604455712 0 0) 0x7f00e0001010 con 0x7f00e40389e0 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.198+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 >> v1:172.21.15.19:6790/0 conn(0x7f00e4030660 legacy=0x7f00e402d3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.199+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 >> v1:172.21.15.19:6789/0 conn(0x7f00e40c8240 legacy=0x7f00e415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.199+0000 7f00daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.199+0000 7f00daffd640 1 monclient: found mon.b 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.199+0000 7f00daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.199+0000 7f00daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.199+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e415ffb0 con 0x7f00e40389e0 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:53.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 10 monclient: _renew_subs 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00e0001920 con 0x7f00e40389e0 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e415c900 con 0x7f00e40389e0 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 10 monclient: _renew_subs 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e4160dc0 con 0x7f00e40389e0 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00eb59e640 1 librados: init done 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:53.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f00e0004a00 con 0x7f00e40389e0 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.200+0000 7f00daffd640 10 monclient: dump: 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:20:53.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.201+0000 7f00e8d99640 4 set_mon_vals no callback set 2024-03-29T11:20:53.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.201+0000 7f00eb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.202+0000 7f00eb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.202+0000 7f00eb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.202+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d0005080 con 0x7f00e40389e0 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.205+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f00e001a9d0 con 0x7f00e40389e0 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.206+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f00e0084950 con 0x7f00e40389e0 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.206+0000 7f00daffd640 10 client.4270.objecter ms_dispatch 0x7f00ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.206+0000 7f00daffd640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.206+0000 7f00daffd640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:53.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.206+0000 7f00daffd640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:20:53.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.206+0000 7f00ea59c640 10 monclient: get_auth_request con 0x7f00cc06dc40 auth_method 0 2024-03-29T11:20:53.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.207+0000 7f00daffd640 10 client.4270.objecter ms_handle_connect 0x7f00cc06dc40 2024-03-29T11:20:53.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.214+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00e0050060 con 0x7f00e40389e0 2024-03-29T11:20:53.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.215+0000 7f00daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:53.191 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.215+0000 7f00daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:53.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.541+0000 7f00eb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:53.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.541+0000 7f00eb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:53.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.541+0000 7f00eb59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:20:53.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.541+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00d0005330 con 0x7f00e40389e0 2024-03-29T11:20:53.519 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:20:53.519 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:53.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.542+0000 7f00daffd640 1 -- 172.21.15.19:0/3733969249 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f00e0052160 con 0x7f00e40389e0 2024-03-29T11:20:53.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.542+0000 7f00daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:53.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.542+0000 7f00daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.545+0000 7f00eb59e640 10 librados: watch_flush enter 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.545+0000 7f00eb59e640 10 librados: watch_flush exit 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.545+0000 7f00eb59e640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.545+0000 7f00eb59e640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.546+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 >> v1:172.21.15.19:6816/3721232097 conn(0x7f00cc06dc40 legacy=0x7f00cc070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.546+0000 7f00eb59e640 10 monclient: shutdown 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.546+0000 7f00eb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:53.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.546+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 >> v1:172.21.15.62:6789/0 conn(0x7f00e40389e0 legacy=0x7f00e41604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:53.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.546+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 shutdown_connections 2024-03-29T11:20:53.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.546+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 >> 172.21.15.19:0/3733969249 conn(0x7f00e4005c00 msgr2=0x7f00e4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:53.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.547+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 shutdown_connections 2024-03-29T11:20:53.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.547+0000 7f00eb59e640 1 -- 172.21.15.19:0/3733969249 wait complete. 2024-03-29T11:20:53.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:20:53.547+0000 7f00eb59e640 1 librados: shutdown 2024-03-29T11:20:53.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:00.348 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:00.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.524+0000 7fbfa304e640 10 monclient: get_monmap_and_config 2024-03-29T11:21:00.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.524+0000 7fbfa304e640 10 monclient: build_initial_monmap 2024-03-29T11:21:00.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.524+0000 7fbfa304e640 10 monclient: monmap: 2024-03-29T11:21:00.500 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:00.524902+0000 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:00.524902+0000 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:00.501 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.525+0000 7fbfa304e640 1 Processor -- start 2024-03-29T11:21:00.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.525+0000 7fbfa304e640 1 -- start start 2024-03-29T11:21:00.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.525+0000 7fbfa304e640 10 monclient: init 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9c1457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9c13f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf9c0a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0aaef0 con 0x7fbf9c13f650 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c145bd0 con 0x7fbf9c1457f0 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c145eb0 con 0x7fbf9c0a9d70 2024-03-29T11:21:00.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:00.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.526+0000 7fbfa304e640 10 monclient(hunting): authenticate will time out at 980.789827s 2024-03-29T11:21:00.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.527+0000 7fbfa0dea640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbf9c1457f0 0x7fbf9c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:54320/0 (socket says 172.21.15.19:54320) 2024-03-29T11:21:00.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.527+0000 7fbfa0dea640 1 -- 172.21.15.19:0/234045226 learned_addr learned my addr 172.21.15.19:0/234045226 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:00.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.529+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 981053547 0 0) 0x7fbf9c145bd0 con 0x7fbf9c1457f0 2024-03-29T11:21:00.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.529+0000 7fbf9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:00.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.529+0000 7fbf9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:00.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.529+0000 7fbf9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:00.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.529+0000 7fbf9b7fe640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:21:00.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.529+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf8c003520 con 0x7fbf9c1457f0 2024-03-29T11:21:00.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023616876 0 0) 0x7fbf9c0aaef0 con 0x7fbf9c13f650 2024-03-29T11:21:00.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:00.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c145bd0 con 0x7fbf9c13f650 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859627111 0 0) 0x7fbf8c003520 con 0x7fbf9c1457f0 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 >> v1:172.21.15.19:6790/0 conn(0x7fbf9c0a9d70 legacy=0x7fbf9c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 >> v1:172.21.15.19:6789/0 conn(0x7fbf9c13f650 legacy=0x7fbf9c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:00.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c146340 con 0x7fbf9c1457f0 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.530+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbf90001690 con 0x7fbf9c1457f0 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: dump: 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.511 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbfa304e640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbfa304e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf900018b0 con 0x7fbf9c1457f0 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 1 -- 172.21.15.19:0/234045226 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbf90001c40 con 0x7fbf9c1457f0 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbf9b7fe640 10 monclient: dump: 2024-03-29T11:21:00.512 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbfa304e640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.531+0000 7fbfa304e640 4 set_mon_vals no callback set 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.532+0000 7fbfa304e640 10 monclient: shutdown 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.532+0000 7fbfa304e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:00.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.532+0000 7fbfa304e640 1 -- 172.21.15.19:0/234045226 >> v1:172.21.15.62:6789/0 conn(0x7fbf9c1457f0 legacy=0x7fbf9c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.532+0000 7fbfa304e640 1 -- 172.21.15.19:0/234045226 shutdown_connections 2024-03-29T11:21:00.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.532+0000 7fbfa304e640 1 -- 172.21.15.19:0/234045226 >> 172.21.15.19:0/234045226 conn(0x7fbf9c066a00 msgr2=0x7fbf9c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:00.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.532+0000 7fbfa304e640 1 -- 172.21.15.19:0/234045226 shutdown_connections 2024-03-29T11:21:00.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.533+0000 7fbfa304e640 1 -- 172.21.15.19:0/234045226 wait complete. 2024-03-29T11:21:00.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 10 monclient: build_initial_monmap 2024-03-29T11:21:00.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 10 monclient: monmap: 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 1 librados: starting msgr at 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 1 librados: starting objecter 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 1 Processor -- start 2024-03-29T11:21:00.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 1 -- start start 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 1 librados: setting wanted keys 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 1 librados: calling monclient init 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.534+0000 7fbfa304e640 10 monclient: init 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.535+0000 7fbfa304e640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.535+0000 7fbfa304e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.535+0000 7fbfa304e640 10 monclient(hunting): picked mon.noname-b con 0x7fbf9c0a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9c13f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9c1457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0adce0 con 0x7fbf9c13f650 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0af450 con 0x7fbf9c0a9d70 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0ad920 con 0x7fbf9c1457f0 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:00.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.536+0000 7fbfa304e640 10 monclient(hunting): authenticate will time out at 980.799204s 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.538+0000 7fbfa0dea640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbf9c0a9d70 0x7fbf9c1d7180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:59372/0 (socket says 172.21.15.19:59372) 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.539+0000 7fbfa0dea640 1 -- 172.21.15.19:0/3483031128 learned_addr learned my addr 172.21.15.19:0/3483031128 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.539+0000 7fbf997fa640 10 client.?.objecter ms_handle_connect 0x7fbf9c13f650 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.539+0000 7fbf997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.539+0000 7fbf997fa640 10 client.?.objecter ms_handle_connect 0x7fbf9c0a9d70 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.539+0000 7fbf997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915982715 0 0) 0x7fbf9c0af450 con 0x7fbf9c0a9d70 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf80004eb0 con 0x7fbf9c0a9d70 2024-03-29T11:21:00.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 10 client.?.objecter ms_handle_connect 0x7fbf9c1457f0 2024-03-29T11:21:00.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.540+0000 7fbf997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:00.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656659522 0 0) 0x7fbf9c0ad920 con 0x7fbf9c1457f0 2024-03-29T11:21:00.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:00.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:00.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:00.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c0af450 con 0x7fbf9c1457f0 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176836828 0 0) 0x7fbf9c0adce0 con 0x7fbf9c13f650 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf9c0ad920 con 0x7fbf9c13f650 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1954561050 0 0) 0x7fbf9c0af450 con 0x7fbf9c1457f0 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.541+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf9c0adce0 con 0x7fbf9c1457f0 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbf94001c30 con 0x7fbf9c1457f0 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:00.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): dump: 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3591185505 0 0) 0x7fbf9c0ad920 con 0x7fbf9c13f650 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf80005b20 con 0x7fbf9c13f650 2024-03-29T11:21:00.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbf88001930 con 0x7fbf9c13f650 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.542+0000 7fbf997fa640 10 monclient(hunting): dump: 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 49829110 0 0) 0x7fbf80004eb0 con 0x7fbf9c0a9d70 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf9c0ad920 con 0x7fbf9c0a9d70 2024-03-29T11:21:00.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2605367847 0 0) 0x7fbf80005b20 con 0x7fbf9c13f650 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 >> v1:172.21.15.19:6790/0 conn(0x7fbf9c1457f0 legacy=0x7fbf9c1e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 >> v1:172.21.15.62:6789/0 conn(0x7fbf9c0a9d70 legacy=0x7fbf9c1d7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 monclient: found mon.a 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c1d7890 con 0x7fbf9c13f650 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 10 monclient: _renew_subs 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:00.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c0ae170 con 0x7fbf9c13f650 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 10 monclient: _renew_subs 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c142240 con 0x7fbf9c13f650 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbfa304e640 1 librados: init done 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf880035f0 con 0x7fbf9c13f650 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.543+0000 7fbf9affd640 4 set_mon_vals no callback set 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.545+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbf88001d80 con 0x7fbf9c13f650 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.545+0000 7fbf997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.545+0000 7fbf997fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.545+0000 7fbf997fa640 10 monclient: dump: 2024-03-29T11:21:00.530 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.545+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fbf880197b0 con 0x7fbf9c13f650 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf86ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf86ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:00.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c0adab0 con 0x7fbf9c13f650 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fbf880836e0 con 0x7fbf9c13f650 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf997fa640 10 client.4296.objecter ms_dispatch 0x7fbf9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf997fa640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf997fa640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbf997fa640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.546+0000 7fbfa0dea640 10 monclient: get_auth_request con 0x7fbf8006e2d0 auth_method 0 2024-03-29T11:21:00.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.550+0000 7fbf997fa640 10 client.4296.objecter ms_handle_connect 0x7fbf8006e2d0 2024-03-29T11:21:00.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.554+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf9c0adab0 con 0x7fbf9c13f650 2024-03-29T11:21:00.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.554+0000 7fbf997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:00.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.554+0000 7fbf997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:00.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.878+0000 7fbf86ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:00.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.878+0000 7fbf86ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:00.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.878+0000 7fbf86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:00.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.878+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf9c0adab0 con 0x7fbf9c13f650 2024-03-29T11:21:00.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.880+0000 7fbf997fa640 1 -- 172.21.15.19:0/3483031128 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fbf9c0adab0 con 0x7fbf9c13f650 2024-03-29T11:21:00.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.880+0000 7fbf997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:00.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.880+0000 7fbf997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:00.857 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:00.857 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.883+0000 7fbf86ffd640 10 librados: watch_flush enter 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.883+0000 7fbf86ffd640 10 librados: watch_flush exit 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.883+0000 7fbf86ffd640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.883+0000 7fbf86ffd640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 >> v1:172.21.15.19:6816/3721232097 conn(0x7fbf8006e2d0 legacy=0x7fbf80070790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 10 monclient: shutdown 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:00.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 >> v1:172.21.15.19:6789/0 conn(0x7fbf9c13f650 legacy=0x7fbf9c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:00.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 shutdown_connections 2024-03-29T11:21:00.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 >> 172.21.15.19:0/3483031128 conn(0x7fbf9c066a00 msgr2=0x7fbf9c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:00.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 shutdown_connections 2024-03-29T11:21:00.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 -- 172.21.15.19:0/3483031128 wait complete. 2024-03-29T11:21:00.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:00.884+0000 7fbf86ffd640 1 librados: shutdown 2024-03-29T11:21:00.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:07.683 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:07.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.853+0000 7f991682d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:07.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.853+0000 7f991682d640 10 monclient: build_initial_monmap 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.853+0000 7f991682d640 10 monclient: monmap: 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:07.854598+0000 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:07.854598+0000 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:07.832 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:07.833 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:07.833 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:07.833 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:07.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.857+0000 7f991682d640 1 Processor -- start 2024-03-29T11:21:07.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.857+0000 7f991682d640 1 -- start start 2024-03-29T11:21:07.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.857+0000 7f991682d640 10 monclient: init 2024-03-29T11:21:07.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:07.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient(hunting): picked mon.noname-a con 0x7f9908038c50 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient(hunting): picked mon.noname-c con 0x7f99080360d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient(hunting): picked mon.noname-b con 0x7f99080c8f80 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9908030b10 con 0x7f9908038c50 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080cc0c0 con 0x7f99080360d0 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9908036540 con 0x7f99080c8f80 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.858+0000 7f991682d640 10 monclient(hunting): authenticate will time out at 988.121637s 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.862+0000 7f991582b640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f9908038c50 0x7f99080cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:45823/0 (socket says 172.21.15.19:45823) 2024-03-29T11:21:07.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.862+0000 7f991582b640 1 -- 172.21.15.19:0/3508554064 learned_addr learned my addr 172.21.15.19:0/3508554064 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:07.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.865+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426561048 0 0) 0x7f9908030b10 con 0x7f9908038c50 2024-03-29T11:21:07.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.865+0000 7f9914829640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.865+0000 7f9914829640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.865+0000 7f9914829640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.865+0000 7f9914829640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.865+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98fc003520 con 0x7f9908038c50 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714034512 0 0) 0x7f98fc003520 con 0x7f9908038c50 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 >> v1:172.21.15.19:6790/0 conn(0x7f99080c8f80 legacy=0x7f99080d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 >> v1:172.21.15.62:6789/0 conn(0x7f99080360d0 legacy=0x7f9908034090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: _finish_hunting 0 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 1 monclient: found mon.noname-a 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080cbc50 con 0x7f9908038c50 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: _finish_auth 0 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: _check_auth_tickets 2024-03-29T11:21:07.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f991004f730 con 0x7f9908038c50 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.866+0000 7f9914829640 10 monclient: dump: 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:07.845 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9910075f10 con 0x7f9908038c50 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f9914829640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f9914829640 1 -- 172.21.15.19:0/3508554064 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f99100758b0 con 0x7f9908038c50 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f9914829640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f9914829640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f9914829640 10 monclient: dump: 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.846 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.847 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.847 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:07.847 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:07.847 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:07.847 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:07.847 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:07.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f991682d640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f991682d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.867+0000 7f991682d640 4 set_mon_vals no callback set 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.868+0000 7f991682d640 10 monclient: shutdown 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.868+0000 7f991682d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.868+0000 7f991682d640 1 -- 172.21.15.19:0/3508554064 >> v1:172.21.15.19:6789/0 conn(0x7f9908038c50 legacy=0x7f99080cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.870+0000 7f991682d640 1 -- 172.21.15.19:0/3508554064 shutdown_connections 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.870+0000 7f991682d640 1 -- 172.21.15.19:0/3508554064 >> 172.21.15.19:0/3508554064 conn(0x7f9908005c00 msgr2=0x7f9908006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.870+0000 7f991682d640 1 -- 172.21.15.19:0/3508554064 shutdown_connections 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.870+0000 7f991682d640 1 -- 172.21.15.19:0/3508554064 wait complete. 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.871+0000 7f991682d640 10 monclient: build_initial_monmap 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.871+0000 7f991682d640 10 monclient: monmap: 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:07.849 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.872+0000 7f991682d640 1 librados: starting msgr at 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.872+0000 7f991682d640 1 librados: starting objecter 2024-03-29T11:21:07.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.872+0000 7f991682d640 1 Processor -- start 2024-03-29T11:21:07.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.874+0000 7f991682d640 1 -- start start 2024-03-29T11:21:07.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.874+0000 7f991682d640 1 librados: setting wanted keys 2024-03-29T11:21:07.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.874+0000 7f991682d640 1 librados: calling monclient init 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.874+0000 7f991682d640 10 monclient: init 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.874+0000 7f991682d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.874+0000 7f991682d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 10 monclient(hunting): picked mon.noname-b con 0x7f99080360d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 10 monclient(hunting): picked mon.noname-a con 0x7f9908038c50 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 10 monclient(hunting): picked mon.noname-c con 0x7f99080c8f80 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080cff10 con 0x7f9908038c50 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99080334f0 con 0x7f99080360d0 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990802d150 con 0x7f99080c8f80 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.875+0000 7f991682d640 10 monclient(hunting): authenticate will time out at 988.138051s 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.879+0000 7f991602c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f99080c8f80 0x7f99080c2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54323/0 (socket says 172.21.15.19:54323) 2024-03-29T11:21:07.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.879+0000 7f991602c640 1 -- 172.21.15.19:0/3030872539 learned_addr learned my addr 172.21.15.19:0/3030872539 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:07.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.881+0000 7f99067fc640 10 client.?.objecter ms_handle_connect 0x7f99080c8f80 2024-03-29T11:21:07.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.881+0000 7f99067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:07.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.881+0000 7f99067fc640 10 client.?.objecter ms_handle_connect 0x7f99080360d0 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.881+0000 7f99067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.881+0000 7f99067fc640 10 client.?.objecter ms_handle_connect 0x7f9908038c50 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497478263 0 0) 0x7f990802d150 con 0x7f99080c8f80 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f4004eb0 con 0x7f99080c8f80 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599173754 0 0) 0x7f99080334f0 con 0x7f99080360d0 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.882+0000 7f99067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990802d150 con 0x7f99080360d0 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2685031602 0 0) 0x7f98f4004eb0 con 0x7f99080c8f80 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99080334f0 con 0x7f99080c8f80 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9900003f60 con 0x7f99080c8f80 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): dump: 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.865 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279655510 0 0) 0x7f99080cff10 con 0x7f9908038c50 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.883+0000 7f99067fc640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:21:07.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98f4004eb0 con 0x7f9908038c50 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2182195702 0 0) 0x7f99080334f0 con 0x7f99080c8f80 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 >> v1:172.21.15.62:6789/0 conn(0x7f99080360d0 legacy=0x7f99080cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 >> v1:172.21.15.19:6789/0 conn(0x7f9908038c50 legacy=0x7f990802d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 monclient: found mon.c 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99080cf170 con 0x7f99080c8f80 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9900000ed0 con 0x7f99080c8f80 2024-03-29T11:21:07.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f99000040c0 con 0x7f99080c8f80 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99077fe640 4 set_mon_vals no callback set 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.884+0000 7f99067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f99067fc640 10 monclient: dump: 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:07.868 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 10 monclient: _renew_subs 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9908033980 con 0x7f99080c8f80 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 10 monclient: _renew_subs 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99080cdfc0 con 0x7f99080c8f80 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.885+0000 7f991682d640 1 librados: init done 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.887+0000 7f991682d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.887+0000 7f991682d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.887+0000 7f991682d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:07.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.887+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9908007b90 con 0x7f99080c8f80 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.889+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f9900001220 con 0x7f99080c8f80 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.890+0000 7f991582b640 10 monclient: get_auth_request con 0x7f98f406dac0 auth_method 0 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.891+0000 7f99067fc640 10 client.4301.objecter ms_handle_connect 0x7f98f406dac0 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.891+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f9900084960 con 0x7f99080c8f80 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.891+0000 7f99067fc640 10 client.4301.objecter ms_dispatch 0x7f9910000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.891+0000 7f99067fc640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.891+0000 7f99067fc640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:07.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.891+0000 7f99067fc640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-29T11:21:07.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.900+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9900050240 con 0x7f99080c8f80 2024-03-29T11:21:07.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.900+0000 7f99067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:07.877 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:07.900+0000 7f99067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:08.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.240+0000 7f991682d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:08.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.240+0000 7f991682d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:08.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.240+0000 7f991682d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:08.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.240+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9908005120 con 0x7f99080c8f80 2024-03-29T11:21:08.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.241+0000 7f99067fc640 1 -- 172.21.15.19:0/3030872539 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f9900052170 con 0x7f99080c8f80 2024-03-29T11:21:08.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.241+0000 7f99067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:08.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.241+0000 7f99067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:08.219 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:08.219 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 10 librados: watch_flush enter 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 10 librados: watch_flush exit 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 10 client.4301.objecter successfully canceled tick 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 >> v1:172.21.15.19:6816/3721232097 conn(0x7f98f406dac0 legacy=0x7f98f406ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 10 monclient: shutdown 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.245+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 >> v1:172.21.15.19:6790/0 conn(0x7f99080c8f80 legacy=0x7f99080c2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.246+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 shutdown_connections 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.246+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 >> 172.21.15.19:0/3030872539 conn(0x7f9908005c00 msgr2=0x7f9908034880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:08.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.247+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 shutdown_connections 2024-03-29T11:21:08.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.249+0000 7f991682d640 1 -- 172.21.15.19:0/3030872539 wait complete. 2024-03-29T11:21:08.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:08.249+0000 7f991682d640 1 librados: shutdown 2024-03-29T11:21:08.242 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:15.050 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.220+0000 7f608d3c8640 10 monclient: get_monmap_and_config 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.220+0000 7f608d3c8640 10 monclient: build_initial_monmap 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.220+0000 7f608d3c8640 10 monclient: monmap: 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:15.222301+0000 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:15.222301+0000 2024-03-29T11:21:15.198 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:15.199 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.199 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:15.199 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:15.199 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:15.199 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.199 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.221+0000 7f608d3c8640 1 Processor -- start 2024-03-29T11:21:15.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 1 -- start start 2024-03-29T11:21:15.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient: init 2024-03-29T11:21:15.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:15.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:15.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient(hunting): picked mon.noname-c con 0x7f60881422b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient(hunting): picked mon.noname-a con 0x7f6088142690 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient(hunting): picked mon.noname-b con 0x7f60880b17d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088142c80 con 0x7f6088142690 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60880aae40 con 0x7f60881422b0 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608813fbd0 con 0x7f60880b17d0 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:15.201 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.222+0000 7f608d3c8640 10 monclient(hunting): authenticate will time out at 995.486985s 2024-03-29T11:21:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.225+0000 7f60867fc640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f6088142690 0x7f60880b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:33889/0 (socket says 172.21.15.19:33889) 2024-03-29T11:21:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.225+0000 7f60867fc640 1 -- 172.21.15.19:0/3971329801 learned_addr learned my addr 172.21.15.19:0/3971329801 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.226+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217149451 0 0) 0x7f6088142c80 con 0x7f6088142690 2024-03-29T11:21:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.226+0000 7f6085ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:15.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.226+0000 7f6085ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.226+0000 7f6085ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.226+0000 7f6085ffb640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:21:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.226+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6070003520 con 0x7f6088142690 2024-03-29T11:21:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.227+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3304846400 0 0) 0x7f6070003520 con 0x7f6088142690 2024-03-29T11:21:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.227+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 >> v1:172.21.15.19:6790/0 conn(0x7f60880b17d0 legacy=0x7f60880b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:15.205 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.228+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 >> v1:172.21.15.62:6789/0 conn(0x7f60881422b0 legacy=0x7f608813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:15.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:15.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:15.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:15.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:21:15.213 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088151940 con 0x7f6088142690 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f608d3c8640 5 monclient: authenticate success, global_id 4305 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f608d3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f607c001690 con 0x7f6088142690 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f6085ffb640 10 monclient: dump: 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.214 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.232+0000 7f608d3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f607c0018b0 con 0x7f6088142690 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f6085ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f608d3c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f608d3c8640 4 set_mon_vals no callback set 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f6085ffb640 1 -- 172.21.15.19:0/3971329801 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f607c001c40 con 0x7f6088142690 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f6085ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:15.215 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f6085ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.233+0000 7f6085ffb640 10 monclient: dump: 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.234+0000 7f608d3c8640 10 monclient: shutdown 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.234+0000 7f608d3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:15.216 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.234+0000 7f608d3c8640 1 -- 172.21.15.19:0/3971329801 >> v1:172.21.15.19:6789/0 conn(0x7f6088142690 legacy=0x7f60880b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:15.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.234+0000 7f608d3c8640 1 -- 172.21.15.19:0/3971329801 shutdown_connections 2024-03-29T11:21:15.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.234+0000 7f608d3c8640 1 -- 172.21.15.19:0/3971329801 >> 172.21.15.19:0/3971329801 conn(0x7f6088066a00 msgr2=0x7f6088066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:15.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.239+0000 7f608d3c8640 1 -- 172.21.15.19:0/3971329801 shutdown_connections 2024-03-29T11:21:15.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.239+0000 7f608d3c8640 1 -- 172.21.15.19:0/3971329801 wait complete. 2024-03-29T11:21:15.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.240+0000 7f608d3c8640 10 monclient: build_initial_monmap 2024-03-29T11:21:15.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.240+0000 7f608d3c8640 10 monclient: monmap: 2024-03-29T11:21:15.218 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:15.218 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.240+0000 7f608d3c8640 1 librados: starting msgr at 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.240+0000 7f608d3c8640 1 librados: starting objecter 2024-03-29T11:21:15.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.240+0000 7f608d3c8640 1 Processor -- start 2024-03-29T11:21:15.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 1 -- start start 2024-03-29T11:21:15.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 1 librados: setting wanted keys 2024-03-29T11:21:15.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 1 librados: calling monclient init 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient: init 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient(hunting): picked mon.noname-a con 0x7f60880b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient(hunting): picked mon.noname-c con 0x7f60881422b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient(hunting): picked mon.noname-b con 0x7f608806fa90 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f608812db10 con 0x7f60880b17d0 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60881d7600 con 0x7f608806fa90 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6088071850 con 0x7f60881422b0 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.241+0000 7f608d3c8640 10 monclient(hunting): authenticate will time out at 995.505824s 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.246+0000 7f60877fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f608806fa90 0x7f60881d6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:45567/0 (socket says 172.21.15.19:45567) 2024-03-29T11:21:15.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.246+0000 7f60877fe640 1 -- 172.21.15.19:0/874270550 learned_addr learned my addr 172.21.15.19:0/874270550 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 10 client.?.objecter ms_handle_connect 0x7f608806fa90 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998004719 0 0) 0x7f60881d7600 con 0x7f608806fa90 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.247+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6074004eb0 con 0x7f608806fa90 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 client.?.objecter ms_handle_connect 0x7f60881422b0 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 client.?.objecter ms_handle_connect 0x7f60880b17d0 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051101309 0 0) 0x7f6088071850 con 0x7f60881422b0 2024-03-29T11:21:15.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.248+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60881d7600 con 0x7f60881422b0 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001924244 0 0) 0x7f608812db10 con 0x7f60880b17d0 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6088071850 con 0x7f60880b17d0 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 139841657 0 0) 0x7f60881d7600 con 0x7f60881422b0 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f608812db10 con 0x7f60881422b0 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f607c001550 con 0x7f60881422b0 2024-03-29T11:21:15.234 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:15.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1159765880 0 0) 0x7f6088071850 con 0x7f60880b17d0 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60881d7600 con 0x7f60880b17d0 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6080001ab0 con 0x7f60880b17d0 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:15.236 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2914641852 0 0) 0x7f60881d7600 con 0x7f60880b17d0 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 >> v1:172.21.15.19:6790/0 conn(0x7f60881422b0 legacy=0x7f60881d67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 >> v1:172.21.15.62:6789/0 conn(0x7f608806fa90 legacy=0x7f60881d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 monclient: found mon.a 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6088071b30 con 0x7f60880b17d0 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f606f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:21:15.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 10 monclient: _renew_subs 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6088071e10 con 0x7f60880b17d0 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 10 monclient: _renew_subs 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60880b73f0 con 0x7f60880b17d0 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.249+0000 7f608d3c8640 1 librados: init done 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6080001fa0 con 0x7f60880b17d0 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6080003b10 con 0x7f60880b17d0 2024-03-29T11:21:15.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 10 monclient: dump: 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:15.239 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f6080019ae0 con 0x7f60880b17d0 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f60857fa640 4 set_mon_vals no callback set 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f6080083af0 con 0x7f60880b17d0 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 10 client.4308.objecter ms_dispatch 0x7f6088000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f608d3c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.252+0000 7f606f7fe640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.253+0000 7f608d3c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.253+0000 7f608d3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.253+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6068005080 con 0x7f60880b17d0 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.253+0000 7f60867fc640 10 monclient: get_auth_request con 0x7f607406e020 auth_method 0 2024-03-29T11:21:15.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.254+0000 7f606f7fe640 10 client.4308.objecter ms_handle_connect 0x7f607406e020 2024-03-29T11:21:15.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.262+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f608004f450 con 0x7f60880b17d0 2024-03-29T11:21:15.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.262+0000 7f606f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:15.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.262+0000 7f606f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:15.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.582+0000 7f608d3c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:15.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.582+0000 7f608d3c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:15.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.582+0000 7f608d3c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:15.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.582+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6068005330 con 0x7f60880b17d0 2024-03-29T11:21:15.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.585+0000 7f606f7fe640 1 -- 172.21.15.19:0/874270550 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f6080051380 con 0x7f60880b17d0 2024-03-29T11:21:15.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.585+0000 7f606f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:15.563 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.585+0000 7f606f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:15.563 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:15.563 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:15.567 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 10 librados: watch_flush enter 2024-03-29T11:21:15.567 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 10 librados: watch_flush exit 2024-03-29T11:21:15.567 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:21:15.567 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:21:15.567 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 >> v1:172.21.15.19:6816/3721232097 conn(0x7f607406e020 legacy=0x7f60740704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 10 monclient: shutdown 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 >> v1:172.21.15.19:6789/0 conn(0x7f60880b17d0 legacy=0x7f60880705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 shutdown_connections 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.589+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 >> 172.21.15.19:0/874270550 conn(0x7f6088066a00 msgr2=0x7f608813c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.590+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 shutdown_connections 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.590+0000 7f608d3c8640 1 -- 172.21.15.19:0/874270550 wait complete. 2024-03-29T11:21:15.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:15.590+0000 7f608d3c8640 1 librados: shutdown 2024-03-29T11:21:15.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:22.392 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:22.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.566+0000 7f4a9b39d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.566+0000 7f4a9b39d640 10 monclient: build_initial_monmap 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.566+0000 7f4a9b39d640 10 monclient: monmap: 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:22.567956+0000 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:22.567956+0000 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:22.544 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.567+0000 7f4a9b39d640 1 Processor -- start 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.567+0000 7f4a9b39d640 1 -- start start 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.567+0000 7f4a9b39d640 10 monclient: init 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:22.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c038c60 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c0324b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c0d11e0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0caf80 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c039850 con 0x7f4a8c0324b0 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0c78d0 con 0x7f4a8c038c60 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:22.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.568+0000 7f4a9b39d640 10 monclient(hunting): authenticate will time out at 1002.832324s 2024-03-29T11:21:22.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.574+0000 7f4a99139640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f4a8c038c60 0x7f4a8c0c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46071/0 (socket says 172.21.15.19:46071) 2024-03-29T11:21:22.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.575+0000 7f4a98938640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4a8c0324b0 0x7f4a8c0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:40865/0 (socket says 172.21.15.19:40865) 2024-03-29T11:21:22.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.575+0000 7f4a98938640 1 -- 172.21.15.19:0/3153349187 learned_addr learned my addr 172.21.15.19:0/3153349187 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:22.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819179150 0 0) 0x7f4a8c0c78d0 con 0x7f4a8c038c60 2024-03-29T11:21:22.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:22.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:22.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:22.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a80003520 con 0x7f4a8c038c60 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790250471 0 0) 0x7f4a8c039850 con 0x7f4a8c0324b0 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:21:22.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.577+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a8c0c78d0 con 0x7f4a8c0324b0 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992671458 0 0) 0x7f4a8c0caf80 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a8c039850 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860688216 0 0) 0x7f4a80003520 con 0x7f4a8c038c60 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 >> v1:172.21.15.62:6789/0 conn(0x7f4a8c0324b0 legacy=0x7f4a8c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 >> v1:172.21.15.19:6789/0 conn(0x7f4a8c0d11e0 legacy=0x7f4a8c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 monclient: found mon.noname-b 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:22.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c0c7a60 con 0x7f4a8c038c60 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4a9404f730 con 0x7f4a8c038c60 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.578+0000 7f4a8bfff640 10 monclient: dump: 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:22.557 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.579+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a94075f10 con 0x7f4a8c038c60 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.579+0000 7f4a8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:22.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.579+0000 7f4a8bfff640 1 -- 172.21.15.19:0/3153349187 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4a940758b0 con 0x7f4a8c038c60 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.579+0000 7f4a8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.579+0000 7f4a8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.579+0000 7f4a8bfff640 10 monclient: dump: 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:22.559 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:22.560 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:22.560 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.581+0000 7f4a9b39d640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:21:22.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.581+0000 7f4a9b39d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:22.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.581+0000 7f4a9b39d640 4 set_mon_vals no callback set 2024-03-29T11:21:22.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 10 monclient: shutdown 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 1 -- 172.21.15.19:0/3153349187 >> v1:172.21.15.19:6790/0 conn(0x7f4a8c038c60 legacy=0x7f4a8c0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 1 -- 172.21.15.19:0/3153349187 shutdown_connections 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 1 -- 172.21.15.19:0/3153349187 >> 172.21.15.19:0/3153349187 conn(0x7f4a8c005c20 msgr2=0x7f4a8c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 1 -- 172.21.15.19:0/3153349187 shutdown_connections 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.582+0000 7f4a9b39d640 1 -- 172.21.15.19:0/3153349187 wait complete. 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.583+0000 7f4a9b39d640 10 monclient: build_initial_monmap 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.583+0000 7f4a9b39d640 10 monclient: monmap: 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:22.561 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:22.562 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:22.562 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.562 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:22.562 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:22.562 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:22.562 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 librados: starting msgr at 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 librados: starting objecter 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 Processor -- start 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 -- start start 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 librados: setting wanted keys 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 librados: calling monclient init 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient: init 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:22.564 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient(hunting): picked mon.noname-a con 0x7f4a8c0324b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient(hunting): picked mon.noname-b con 0x7f4a8c038c60 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient(hunting): picked mon.noname-c con 0x7f4a8c0d11e0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0366b0 con 0x7f4a8c0324b0 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c034720 con 0x7f4a8c038c60 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a8c0359a0 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.584+0000 7f4a9b39d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:22.565 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.585+0000 7f4a9b39d640 10 monclient(hunting): authenticate will time out at 1002.849007s 2024-03-29T11:21:22.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.591+0000 7f4a98938640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4a8c038c60 0x7f4a8c169830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:40969/0 (socket says 172.21.15.19:40969) 2024-03-29T11:21:22.568 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.591+0000 7f4a98938640 1 -- 172.21.15.19:0/2348187870 learned_addr learned my addr 172.21.15.19:0/2348187870 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:22.574 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 client.?.objecter ms_handle_connect 0x7f4a8c038c60 2024-03-29T11:21:22.574 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 client.?.objecter ms_handle_connect 0x7f4a8c0d11e0 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160775708 0 0) 0x7f4a8c0359a0 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a78004eb0 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 client.?.objecter ms_handle_connect 0x7f4a8c0324b0 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.594+0000 7f4a89ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381043885 0 0) 0x7f4a8c034720 con 0x7f4a8c038c60 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:22.575 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a8c0359a0 con 0x7f4a8c038c60 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265201399 0 0) 0x7f4a78004eb0 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a8c034720 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409327819 0 0) 0x7f4a8c0366b0 con 0x7f4a8c0324b0 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a78004eb0 con 0x7f4a8c0324b0 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4a90001850 con 0x7f4a8c0d11e0 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:22.576 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3263182314 0 0) 0x7f4a8c0359a0 con 0x7f4a8c038c60 2024-03-29T11:21:22.577 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a8c0366b0 con 0x7f4a8c038c60 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4a84001800 con 0x7f4a8c038c60 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.595+0000 7f4a89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.596+0000 7f4a89ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.596+0000 7f4a89ffb640 10 monclient(hunting): dump: 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:22.578 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:22.579 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3238427440 0 0) 0x7f4a8c0366b0 con 0x7f4a8c038c60 2024-03-29T11:21:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 >> v1:172.21.15.19:6790/0 conn(0x7f4a8c0d11e0 legacy=0x7f4a8c169f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 >> v1:172.21.15.19:6789/0 conn(0x7f4a8c0324b0 legacy=0x7f4a8c035290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 1 monclient: found mon.b 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.599+0000 7f4a89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8c036900 con 0x7f4a8c038c60 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 10 monclient: _renew_subs 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8c037000 con 0x7f4a8c038c60 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:22.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 10 monclient: _renew_subs 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a8c175140 con 0x7f4a8c038c60 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.600+0000 7f4a9b39d640 1 librados: init done 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.601+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a84001d10 con 0x7f4a8c038c60 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.601+0000 7f4a89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.601+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4a84004980 con 0x7f4a8c038c60 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.601+0000 7f4a89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.601+0000 7f4a89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:22.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.601+0000 7f4a89ffb640 10 monclient: dump: 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.602+0000 7f4a9b39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:22.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.602+0000 7f4a9b39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:22.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.602+0000 7f4a9b39d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:22.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.602+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a8c007b70 con 0x7f4a8c038c60 2024-03-29T11:21:22.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.602+0000 7f4a8affd640 4 set_mon_vals no callback set 2024-03-29T11:21:22.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f4a84004ae0 con 0x7f4a8c038c60 2024-03-29T11:21:22.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f4a84084a30 con 0x7f4a8c038c60 2024-03-29T11:21:22.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a89ffb640 10 client.4288.objecter ms_dispatch 0x7f4a94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:22.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a89ffb640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:22.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a89ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:22.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a89ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T11:21:22.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.606+0000 7f4a99139640 10 monclient: get_auth_request con 0x7f4a7806dcf0 auth_method 0 2024-03-29T11:21:22.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.607+0000 7f4a89ffb640 10 client.4288.objecter ms_handle_connect 0x7f4a7806dcf0 2024-03-29T11:21:22.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.609+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a84050140 con 0x7f4a8c038c60 2024-03-29T11:21:22.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.609+0000 7f4a89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:22.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.609+0000 7f4a89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:22.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.937+0000 7f4a9b39d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:22.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.937+0000 7f4a9b39d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:22.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.937+0000 7f4a9b39d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:22.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.937+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a8c03b750 con 0x7f4a8c038c60 2024-03-29T11:21:22.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.939+0000 7f4a89ffb640 1 -- 172.21.15.19:0/2348187870 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f4a84052240 con 0x7f4a8c038c60 2024-03-29T11:21:22.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.939+0000 7f4a89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:22.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.939+0000 7f4a89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:22.917 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:22.917 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:22.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.942+0000 7f4a9b39d640 10 librados: watch_flush enter 2024-03-29T11:21:22.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 10 librados: watch_flush exit 2024-03-29T11:21:22.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T11:21:22.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 10 client.4288.objecter successfully canceled tick 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 >> v1:172.21.15.19:6816/3721232097 conn(0x7f4a7806dcf0 legacy=0x7f4a780701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 10 monclient: shutdown 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 >> v1:172.21.15.62:6789/0 conn(0x7f4a8c038c60 legacy=0x7f4a8c169830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 shutdown_connections 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.943+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 >> 172.21.15.19:0/2348187870 conn(0x7f4a8c005c20 msgr2=0x7f4a8c006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.944+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 shutdown_connections 2024-03-29T11:21:22.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.944+0000 7f4a9b39d640 1 -- 172.21.15.19:0/2348187870 wait complete. 2024-03-29T11:21:22.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:22.944+0000 7f4a9b39d640 1 librados: shutdown 2024-03-29T11:21:22.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:29.746 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:29.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.926+0000 7f90da593640 10 monclient: get_monmap_and_config 2024-03-29T11:21:29.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.926+0000 7f90da593640 10 monclient: build_initial_monmap 2024-03-29T11:21:29.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.926+0000 7f90da593640 10 monclient: monmap: 2024-03-29T11:21:29.903 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:29.903 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.903 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:29.927847+0000 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:29.927847+0000 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:29.904 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.930+0000 7f90da593640 1 Processor -- start 2024-03-29T11:21:29.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.930+0000 7f90da593640 1 -- start start 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.930+0000 7f90da593640 10 monclient: init 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.931+0000 7f90da593640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.931+0000 7f90da593640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.931+0000 7f90da593640 10 monclient(hunting): picked mon.noname-c con 0x7f90d41422b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.931+0000 7f90da593640 10 monclient(hunting): picked mon.noname-b con 0x7f90d4142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.931+0000 7f90da593640 10 monclient(hunting): picked mon.noname-a con 0x7f90d40b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.932+0000 7f90da593640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d4142c80 con 0x7f90d40b17d0 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.932+0000 7f90da593640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d40aae40 con 0x7f90d41422b0 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.932+0000 7f90da593640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d413fbd0 con 0x7f90d4142690 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.932+0000 7f90da593640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:29.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.932+0000 7f90da593640 10 monclient(hunting): authenticate will time out at 1010.196055s 2024-03-29T11:21:29.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.937+0000 7f90cbfff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90d4142690 0x7f90d40b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60947/0 (socket says 172.21.15.19:60947) 2024-03-29T11:21:29.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.937+0000 7f90cbfff640 1 -- 172.21.15.19:0/3809874616 learned_addr learned my addr 172.21.15.19:0/3809874616 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968052168 0 0) 0x7f90d413fbd0 con 0x7f90d4142690 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c0003520 con 0x7f90d4142690 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3004209229 0 0) 0x7f90c0003520 con 0x7f90d4142690 2024-03-29T11:21:29.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 >> v1:172.21.15.62:6789/0 conn(0x7f90d41422b0 legacy=0x7f90d413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 >> v1:172.21.15.19:6789/0 conn(0x7f90d40b17d0 legacy=0x7f90d40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 monclient: found mon.noname-b 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d4151940 con 0x7f90d4142690 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:29.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90c4001690 con 0x7f90d4142690 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.938+0000 7f90d37fe640 10 monclient: dump: 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.917 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.939+0000 7f90da593640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:21:29.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.939+0000 7f90da593640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90c40017f0 con 0x7f90d4142690 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90d37fe640 1 -- 172.21.15.19:0/3809874616 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90c4001bc0 con 0x7f90d4142690 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90d37fe640 10 monclient: dump: 2024-03-29T11:21:29.919 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90da593640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:29.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.941+0000 7f90da593640 4 set_mon_vals no callback set 2024-03-29T11:21:29.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.942+0000 7f90da593640 10 monclient: shutdown 2024-03-29T11:21:29.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.942+0000 7f90da593640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:29.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.942+0000 7f90da593640 1 -- 172.21.15.19:0/3809874616 >> v1:172.21.15.19:6790/0 conn(0x7f90d4142690 legacy=0x7f90d40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.943+0000 7f90da593640 1 -- 172.21.15.19:0/3809874616 shutdown_connections 2024-03-29T11:21:29.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.943+0000 7f90da593640 1 -- 172.21.15.19:0/3809874616 >> 172.21.15.19:0/3809874616 conn(0x7f90d4066a00 msgr2=0x7f90d4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:29.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.943+0000 7f90da593640 1 -- 172.21.15.19:0/3809874616 shutdown_connections 2024-03-29T11:21:29.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.943+0000 7f90da593640 1 -- 172.21.15.19:0/3809874616 wait complete. 2024-03-29T11:21:29.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 10 monclient: build_initial_monmap 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 10 monclient: monmap: 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 1 librados: starting msgr at 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 1 librados: starting objecter 2024-03-29T11:21:29.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 1 Processor -- start 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 1 -- start start 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 1 librados: setting wanted keys 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 1 librados: calling monclient init 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.944+0000 7f90da593640 10 monclient: init 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.945+0000 7f90da593640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.945+0000 7f90da593640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 10 monclient(hunting): picked mon.noname-a con 0x7f90d40b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 10 monclient(hunting): picked mon.noname-c con 0x7f90d41422b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 10 monclient(hunting): picked mon.noname-b con 0x7f90d4142690 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d406edf0 con 0x7f90d40b17d0 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d406fb20 con 0x7f90d4142690 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d406fe00 con 0x7f90d41422b0 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:29.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.946+0000 7f90da593640 10 monclient(hunting): authenticate will time out at 1010.210340s 2024-03-29T11:21:29.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.951+0000 7f90cbfff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90d41422b0 0x7f90d4070700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:32841/0 (socket says 172.21.15.19:32841) 2024-03-29T11:21:29.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.951+0000 7f90cbfff640 1 -- 172.21.15.19:0/3078847208 learned_addr learned my addr 172.21.15.19:0/3078847208 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:29.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.953+0000 7f90d17fa640 10 client.?.objecter ms_handle_connect 0x7f90d41422b0 2024-03-29T11:21:29.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.953+0000 7f90d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:29.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.953+0000 7f90d17fa640 10 client.?.objecter ms_handle_connect 0x7f90d40b17d0 2024-03-29T11:21:29.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.953+0000 7f90d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:29.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 client.?.objecter ms_handle_connect 0x7f90d4142690 2024-03-29T11:21:29.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357485806 0 0) 0x7f90d406fe00 con 0x7f90d41422b0 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b4004eb0 con 0x7f90d41422b0 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149584964 0 0) 0x7f90d406fb20 con 0x7f90d4142690 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d406fe00 con 0x7f90d4142690 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460625028 0 0) 0x7f90d406edf0 con 0x7f90d40b17d0 2024-03-29T11:21:29.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:29.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:29.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:29.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:21:29.933 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.954+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d406fb20 con 0x7f90d40b17d0 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994650706 0 0) 0x7f90b4004eb0 con 0x7f90d41422b0 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d406edf0 con 0x7f90d41422b0 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3746366308 0 0) 0x7f90d406fb20 con 0x7f90d40b17d0 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90b4004eb0 con 0x7f90d40b17d0 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90c4001650 con 0x7f90d41422b0 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:29.934 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90bc001700 con 0x7f90d40b17d0 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:29.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3111097156 0 0) 0x7f90d406fe00 con 0x7f90d4142690 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90d406fb20 con 0x7f90d4142690 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90cc001c30 con 0x7f90d4142690 2024-03-29T11:21:29.936 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.955+0000 7f90d17fa640 10 monclient(hunting): dump: 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.937 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 810022854 0 0) 0x7f90d406fb20 con 0x7f90d4142690 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 >> v1:172.21.15.19:6790/0 conn(0x7f90d41422b0 legacy=0x7f90d4070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 >> v1:172.21.15.19:6789/0 conn(0x7f90d40b17d0 legacy=0x7f90d41e6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 1 monclient: found mon.b 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d40700e0 con 0x7f90d4142690 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90da593640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90da593640 10 monclient: _renew_subs 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90da593640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:29.938 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.956+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d41e7430 con 0x7f90d4142690 2024-03-29T11:21:29.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90cc002040 con 0x7f90d4142690 2024-03-29T11:21:29.940 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90cc003b10 con 0x7f90d4142690 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90d17fa640 10 monclient: dump: 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:29.941 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90da593640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90da593640 10 monclient: _renew_subs 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90da593640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d41327a0 con 0x7f90d4142690 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.957+0000 7f90da593640 1 librados: init done 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.958+0000 7f90d27fc640 4 set_mon_vals no callback set 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.960+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f90cc019ae0 con 0x7f90d4142690 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f90cc083970 con 0x7f90d4142690 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90d17fa640 10 client.4294.objecter ms_dispatch 0x7f90d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90d17fa640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90d17fa640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90d17fa640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-29T11:21:29.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90da593640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:29.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90da593640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:29.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90da593640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:29.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.961+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b8005080 con 0x7f90d4142690 2024-03-29T11:21:29.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.962+0000 7f90d3fff640 10 monclient: get_auth_request con 0x7f90b406e380 auth_method 0 2024-03-29T11:21:29.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.963+0000 7f90d17fa640 10 client.4294.objecter ms_handle_connect 0x7f90b406e380 2024-03-29T11:21:29.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.970+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90cc04f250 con 0x7f90d4142690 2024-03-29T11:21:29.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.970+0000 7f90d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:29.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:29.970+0000 7f90d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:30.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.310+0000 7f90da593640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:30.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.310+0000 7f90da593640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:30.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.310+0000 7f90da593640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:30.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.310+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90b8005330 con 0x7f90d4142690 2024-03-29T11:21:30.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.311+0000 7f90d17fa640 1 -- 172.21.15.19:0/3078847208 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f90cc051180 con 0x7f90d4142690 2024-03-29T11:21:30.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.311+0000 7f90d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:30.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.311+0000 7f90d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:30.289 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:30.289 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:30.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.315+0000 7f90da593640 10 librados: watch_flush enter 2024-03-29T11:21:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 10 librados: watch_flush exit 2024-03-29T11:21:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-29T11:21:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 10 client.4294.objecter successfully canceled tick 2024-03-29T11:21:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 >> v1:172.21.15.19:6816/3721232097 conn(0x7f90b406e380 legacy=0x7f90b4070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:30.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 10 monclient: shutdown 2024-03-29T11:21:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 >> v1:172.21.15.62:6789/0 conn(0x7f90d4142690 legacy=0x7f90d4070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 shutdown_connections 2024-03-29T11:21:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.317+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 >> 172.21.15.19:0/3078847208 conn(0x7f90d4066a00 msgr2=0x7f90d413df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.318+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 shutdown_connections 2024-03-29T11:21:30.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.318+0000 7f90da593640 1 -- 172.21.15.19:0/3078847208 wait complete. 2024-03-29T11:21:30.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:30.318+0000 7f90da593640 1 librados: shutdown 2024-03-29T11:21:30.315 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:37.122 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:37.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.298+0000 7f75de2d6640 10 monclient: get_monmap_and_config 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.298+0000 7f75de2d6640 10 monclient: build_initial_monmap 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.298+0000 7f75de2d6640 10 monclient: monmap: 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:37.299403+0000 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:37.299403+0000 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:37.275 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:37.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.300+0000 7f75de2d6640 1 Processor -- start 2024-03-29T11:21:37.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.300+0000 7f75de2d6640 1 -- start start 2024-03-29T11:21:37.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.300+0000 7f75de2d6640 10 monclient: init 2024-03-29T11:21:37.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:37.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient(hunting): picked mon.noname-b con 0x7f75d81422b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient(hunting): picked mon.noname-c con 0x7f75d8142690 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient(hunting): picked mon.noname-a con 0x7f75d80b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d8142c80 con 0x7f75d80b17d0 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d80aae40 con 0x7f75d8142690 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d813fbd0 con 0x7f75d81422b0 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:37.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.301+0000 7f75de2d6640 10 monclient(hunting): authenticate will time out at 1017.564830s 2024-03-29T11:21:37.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.306+0000 7f75d77fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f75d8142690 0x7f75d80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:57321/0 (socket says 172.21.15.19:57321) 2024-03-29T11:21:37.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.306+0000 7f75d77fe640 1 -- 172.21.15.19:0/2456242092 learned_addr learned my addr 172.21.15.19:0/2456242092 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:37.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.307+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077885759 0 0) 0x7f75d80aae40 con 0x7f75d8142690 2024-03-29T11:21:37.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.307+0000 7f75d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:37.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.307+0000 7f75d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:37.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.307+0000 7f75d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:37.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.307+0000 7f75d6ffd640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:21:37.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.307+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c0003520 con 0x7f75d8142690 2024-03-29T11:21:37.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.308+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640178155 0 0) 0x7f75d8142c80 con 0x7f75d80b17d0 2024-03-29T11:21:37.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.308+0000 7f75d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:37.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.308+0000 7f75d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:37.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.308+0000 7f75d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:37.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.308+0000 7f75d6ffd640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.308+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d80aae40 con 0x7f75d80b17d0 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3723388754 0 0) 0x7f75c0003520 con 0x7f75d8142690 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 >> v1:172.21.15.19:6790/0 conn(0x7f75d81422b0 legacy=0x7f75d813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 >> v1:172.21.15.19:6789/0 conn(0x7f75d80b17d0 legacy=0x7f75d80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:37.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8151940 con 0x7f75d8142690 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f75cc001610 con 0x7f75d8142690 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: dump: 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.287 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75de2d6640 5 monclient: authenticate success, global_id 4297 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75de2d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75cc001770 con 0x7f75d8142690 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 1 -- 172.21.15.19:0/2456242092 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f75cc001bc0 con 0x7f75d8142690 2024-03-29T11:21:37.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75d6ffd640 10 monclient: dump: 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75de2d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:37.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.309+0000 7f75de2d6640 4 set_mon_vals no callback set 2024-03-29T11:21:37.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.310+0000 7f75de2d6640 10 monclient: shutdown 2024-03-29T11:21:37.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.310+0000 7f75de2d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:37.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.310+0000 7f75de2d6640 1 -- 172.21.15.19:0/2456242092 >> v1:172.21.15.62:6789/0 conn(0x7f75d8142690 legacy=0x7f75d80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:37.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.311+0000 7f75de2d6640 1 -- 172.21.15.19:0/2456242092 shutdown_connections 2024-03-29T11:21:37.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.311+0000 7f75de2d6640 1 -- 172.21.15.19:0/2456242092 >> 172.21.15.19:0/2456242092 conn(0x7f75d8066a00 msgr2=0x7f75d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:37.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.313+0000 7f75de2d6640 1 -- 172.21.15.19:0/2456242092 shutdown_connections 2024-03-29T11:21:37.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.313+0000 7f75de2d6640 1 -- 172.21.15.19:0/2456242092 wait complete. 2024-03-29T11:21:37.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.314+0000 7f75de2d6640 10 monclient: build_initial_monmap 2024-03-29T11:21:37.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.314+0000 7f75de2d6640 10 monclient: monmap: 2024-03-29T11:21:37.292 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:37.292 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:37.292 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.314+0000 7f75de2d6640 1 librados: starting msgr at 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.314+0000 7f75de2d6640 1 librados: starting objecter 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.315+0000 7f75de2d6640 1 Processor -- start 2024-03-29T11:21:37.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.315+0000 7f75de2d6640 1 -- start start 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.315+0000 7f75de2d6640 1 librados: setting wanted keys 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.315+0000 7f75de2d6640 1 librados: calling monclient init 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.315+0000 7f75de2d6640 10 monclient: init 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient(hunting): picked mon.noname-c con 0x7f75d80b17d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient(hunting): picked mon.noname-a con 0x7f75d81422b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient(hunting): picked mon.noname-b con 0x7f75d8142690 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d806f320 con 0x7f75d81422b0 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d806f040 con 0x7f75d8142690 2024-03-29T11:21:37.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d806de90 con 0x7f75d80b17d0 2024-03-29T11:21:37.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:37.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.316+0000 7f75de2d6640 10 monclient(hunting): authenticate will time out at 1017.579191s 2024-03-29T11:21:37.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.323+0000 7f75d7fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f75d80b17d0 0x7f75d812bef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52469/0 (socket says 172.21.15.19:52469) 2024-03-29T11:21:37.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.323+0000 7f75d7fff640 1 -- 172.21.15.19:0/1629100908 learned_addr learned my addr 172.21.15.19:0/1629100908 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:37.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 client.?.objecter ms_handle_connect 0x7f75d8142690 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 client.?.objecter ms_handle_connect 0x7f75d81422b0 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 client.?.objecter ms_handle_connect 0x7f75d80b17d0 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217080517 0 0) 0x7f75d806f040 con 0x7f75d8142690 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4004eb0 con 0x7f75d8142690 2024-03-29T11:21:37.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719273155 0 0) 0x7f75d806f320 con 0x7f75d81422b0 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d806f040 con 0x7f75d81422b0 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.326+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597118895 0 0) 0x7f75c4004eb0 con 0x7f75d8142690 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75d806f320 con 0x7f75d8142690 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f75d0001cb0 con 0x7f75d8142690 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient(hunting): dump: 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:37.305 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1450765390 0 0) 0x7f75d806f320 con 0x7f75d8142690 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 >> v1:172.21.15.19:6790/0 conn(0x7f75d80b17d0 legacy=0x7f75d812bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 >> v1:172.21.15.19:6789/0 conn(0x7f75d81422b0 legacy=0x7f75d81eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:21:37.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 monclient: found mon.b 2024-03-29T11:21:37.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:37.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:37.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d806e1c0 con 0x7f75d8142690 2024-03-29T11:21:37.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:21:37.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:21:37.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.327+0000 7f75d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:37.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.328+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d0003990 con 0x7f75d8142690 2024-03-29T11:21:37.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.328+0000 7f75d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:37.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.328+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f75d0003d60 con 0x7f75d8142690 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.328+0000 7f75d67fc640 4 set_mon_vals no callback set 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.329+0000 7f75d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.329+0000 7f75d4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.329+0000 7f75d4ff9640 10 monclient: dump: 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:37.309 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 10 monclient: _renew_subs 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8131a90 con 0x7f75d8142690 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 10 monclient: _renew_subs 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d8151100 con 0x7f75d8142690 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.330+0000 7f75de2d6640 1 librados: init done 2024-03-29T11:21:37.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.331+0000 7f75de2d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:37.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.331+0000 7f75de2d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:37.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.331+0000 7f75de2d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:37.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.331+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d806e020 con 0x7f75d8142690 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f75d0003ec0 con 0x7f75d8142690 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f75d00838a0 con 0x7f75d8142690 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d4ff9640 10 client.4300.objecter ms_dispatch 0x7f75d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d4ff9640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d4ff9640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d4ff9640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.333+0000 7f75d7fff640 10 monclient: get_auth_request con 0x7f75c406dd90 auth_method 0 2024-03-29T11:21:37.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.334+0000 7f75d4ff9640 10 client.4300.objecter ms_handle_connect 0x7f75c406dd90 2024-03-29T11:21:37.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.342+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75d004f180 con 0x7f75d8142690 2024-03-29T11:21:37.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.342+0000 7f75d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:37.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.342+0000 7f75d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:37.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.683+0000 7f75de2d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.683+0000 7f75de2d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.683+0000 7f75de2d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.683+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75d806e020 con 0x7f75d8142690 2024-03-29T11:21:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.684+0000 7f75d4ff9640 1 -- 172.21.15.19:0/1629100908 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f75d806e020 con 0x7f75d8142690 2024-03-29T11:21:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.684+0000 7f75d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:37.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.684+0000 7f75d4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:37.661 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:37.661 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:37.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 10 librados: watch_flush enter 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 10 librados: watch_flush exit 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 >> v1:172.21.15.19:6816/3721232097 conn(0x7f75c406dd90 legacy=0x7f75c4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 10 monclient: shutdown 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:37.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.688+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 >> v1:172.21.15.62:6789/0 conn(0x7f75d8142690 legacy=0x7f75d81eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.689+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 shutdown_connections 2024-03-29T11:21:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.689+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 >> 172.21.15.19:0/1629100908 conn(0x7f75d8066a00 msgr2=0x7f75d813d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.689+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 shutdown_connections 2024-03-29T11:21:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.689+0000 7f75de2d6640 1 -- 172.21.15.19:0/1629100908 wait complete. 2024-03-29T11:21:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:37.689+0000 7f75de2d6640 1 librados: shutdown 2024-03-29T11:21:37.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:44.487 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:44.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.661+0000 7f6b04723640 10 monclient: get_monmap_and_config 2024-03-29T11:21:44.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.661+0000 7f6b04723640 10 monclient: build_initial_monmap 2024-03-29T11:21:44.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.661+0000 7f6b04723640 10 monclient: monmap: 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:44.661838+0000 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:44.661838+0000 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:44.638 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.662+0000 7f6b04723640 1 Processor -- start 2024-03-29T11:21:44.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.662+0000 7f6b04723640 1 -- start start 2024-03-29T11:21:44.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.662+0000 7f6b04723640 10 monclient: init 2024-03-29T11:21:44.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:44.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:44.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient(hunting): picked mon.noname-b con 0x7f6afc06cac0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient(hunting): picked mon.noname-c con 0x7f6afc06cea0 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient(hunting): picked mon.noname-a con 0x7f6afc145e80 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc0ac850 con 0x7f6afc145e80 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc070470 con 0x7f6afc06cea0 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc072600 con 0x7f6afc06cac0 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.663+0000 7f6b04723640 10 monclient(hunting): authenticate will time out at 1024.926836s 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.664+0000 7f6b024bf640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f6afc06cac0 0x7f6afc071ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:47926/0 (socket says 172.21.15.19:47926) 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.664+0000 7f6b024bf640 1 -- 172.21.15.19:0/2335549772 learned_addr learned my addr 172.21.15.19:0/2335549772 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272028539 0 0) 0x7f6afc0ac850 con 0x7f6afc145e80 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:44.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:44.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:21:44.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae8003520 con 0x7f6afc145e80 2024-03-29T11:21:44.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948185682 0 0) 0x7f6afc070470 con 0x7f6afc06cea0 2024-03-29T11:21:44.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6afc0ac850 con 0x7f6afc06cea0 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392209978 0 0) 0x7f6afc072600 con 0x7f6afc06cac0 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:21:44.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6afc070470 con 0x7f6afc06cac0 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1766783626 0 0) 0x7f6ae8003520 con 0x7f6afc145e80 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 >> v1:172.21.15.19:6790/0 conn(0x7f6afc06cac0 legacy=0x7f6afc071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 >> v1:172.21.15.62:6789/0 conn(0x7f6afc06cea0 legacy=0x7f6afc145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 1 monclient: found mon.noname-a 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.665+0000 7f6b014bd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc1462d0 con 0x7f6afc145e80 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 10 monclient: _finish_auth 0 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:44.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6af4001a10 con 0x7f6afc145e80 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b014bd640 10 monclient: dump: 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:44.648 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b04723640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.666+0000 7f6b04723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6af4001c60 con 0x7f6afc145e80 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b014bd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b04723640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b04723640 4 set_mon_vals no callback set 2024-03-29T11:21:44.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b014bd640 1 -- 172.21.15.19:0/2335549772 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6af4001ff0 con 0x7f6afc145e80 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b014bd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b014bd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.667+0000 7f6b014bd640 10 monclient: dump: 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:44.650 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.651 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:44.651 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:44.651 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:44.651 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.668+0000 7f6b04723640 10 monclient: shutdown 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.668+0000 7f6b04723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.668+0000 7f6b04723640 1 -- 172.21.15.19:0/2335549772 >> v1:172.21.15.19:6789/0 conn(0x7f6afc145e80 legacy=0x7f6afc14db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.668+0000 7f6b04723640 1 -- 172.21.15.19:0/2335549772 shutdown_connections 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.668+0000 7f6b04723640 1 -- 172.21.15.19:0/2335549772 >> 172.21.15.19:0/2335549772 conn(0x7f6afc0675b0 msgr2=0x7f6afc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.669+0000 7f6b04723640 1 -- 172.21.15.19:0/2335549772 shutdown_connections 2024-03-29T11:21:44.656 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.669+0000 7f6b04723640 1 -- 172.21.15.19:0/2335549772 wait complete. 2024-03-29T11:21:44.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 10 monclient: build_initial_monmap 2024-03-29T11:21:44.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 10 monclient: monmap: 2024-03-29T11:21:44.657 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:44.657 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:44.657 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:44.657 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 1 librados: starting msgr at 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 1 librados: starting objecter 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 1 Processor -- start 2024-03-29T11:21:44.658 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 1 -- start start 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 1 librados: setting wanted keys 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 1 librados: calling monclient init 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.670+0000 7f6b04723640 10 monclient: init 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.671+0000 7f6b04723640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.671+0000 7f6b04723640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 10 monclient(hunting): picked mon.noname-a con 0x7f6afc06cac0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 10 monclient(hunting): picked mon.noname-c con 0x7f6afc06cea0 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 10 monclient(hunting): picked mon.noname-b con 0x7f6afc145e80 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b01cbe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f6afc06cea0 0x7f6afc11ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48284/0 (socket says 172.21.15.19:48284) 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b01cbe640 1 -- 172.21.15.19:0/1519946902 learned_addr learned my addr 172.21.15.19:0/1519946902 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc1420d0 con 0x7f6afc06cac0 2024-03-29T11:21:44.659 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6aeeffd640 10 client.?.objecter ms_handle_connect 0x7f6afc145e80 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc142b80 con 0x7f6afc145e80 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6afc142df0 con 0x7f6afc06cea0 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.672+0000 7f6b04723640 10 monclient(hunting): authenticate will time out at 1024.935911s 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 client.?.objecter ms_handle_connect 0x7f6afc06cea0 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 client.?.objecter ms_handle_connect 0x7f6afc06cac0 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273580068 0 0) 0x7f6afc1420d0 con 0x7f6afc06cac0 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:44.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ae0004eb0 con 0x7f6afc06cac0 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335881966 0 0) 0x7f6afc142b80 con 0x7f6afc145e80 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6afc1420d0 con 0x7f6afc145e80 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266485964 0 0) 0x7f6afc142df0 con 0x7f6afc06cea0 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:44.661 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.673+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6afc142b80 con 0x7f6afc06cea0 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1216734722 0 0) 0x7f6ae0004eb0 con 0x7f6afc06cac0 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6afc142df0 con 0x7f6afc06cac0 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6af0001910 con 0x7f6afc06cac0 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 10 monclient(hunting): dump: 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.662 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.674+0000 7f6aeeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2011322186 0 0) 0x7f6afc1420d0 con 0x7f6afc145e80 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ae0004eb0 con 0x7f6afc145e80 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2328625547 0 0) 0x7f6afc142b80 con 0x7f6afc06cea0 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6afc1420d0 con 0x7f6afc06cea0 2024-03-29T11:21:44.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6af4001dd0 con 0x7f6afc145e80 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient(hunting): dump: 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:44.664 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2330104903 0 0) 0x7f6afc142df0 con 0x7f6afc06cac0 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 >> v1:172.21.15.19:6790/0 conn(0x7f6afc06cea0 legacy=0x7f6afc11ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 >> v1:172.21.15.62:6789/0 conn(0x7f6afc145e80 legacy=0x7f6afc0ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 monclient: found mon.a 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6afc1430d0 con 0x7f6afc06cac0 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:44.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 10 monclient: _renew_subs 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6afc11d7a0 con 0x7f6afc06cac0 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 10 monclient: _renew_subs 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6afc1206b0 con 0x7f6afc06cac0 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6b04723640 1 librados: init done 2024-03-29T11:21:44.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6af00047e0 con 0x7f6afc06cac0 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.675+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6af0004af0 con 0x7f6afc06cac0 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.676+0000 7f6aeeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.676+0000 7f6aeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.676+0000 7f6aeeffd640 10 monclient: dump: 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:44.667 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.676+0000 7f6b00cbc640 4 set_mon_vals no callback set 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.677+0000 7f6b04723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.677+0000 7f6b04723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.677+0000 7f6b04723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.677+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae4005080 con 0x7f6afc06cac0 2024-03-29T11:21:44.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.679+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f6af001aac0 con 0x7f6afc06cac0 2024-03-29T11:21:44.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.680+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f6af00848a0 con 0x7f6afc06cac0 2024-03-29T11:21:44.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.680+0000 7f6aeeffd640 10 client.4332.objecter ms_dispatch 0x7f6afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:44.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.680+0000 7f6aeeffd640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:44.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.680+0000 7f6aeeffd640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:44.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.681+0000 7f6aeeffd640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T11:21:44.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.681+0000 7f6b01cbe640 10 monclient: get_auth_request con 0x7f6ae006e370 auth_method 0 2024-03-29T11:21:44.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.681+0000 7f6aeeffd640 10 client.4332.objecter ms_handle_connect 0x7f6ae006e370 2024-03-29T11:21:44.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.685+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6af0050180 con 0x7f6afc06cac0 2024-03-29T11:21:44.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.686+0000 7f6aeeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:44.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:44.686+0000 7f6aeeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:45.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.050+0000 7f6b04723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:45.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.050+0000 7f6b04723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:45.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.050+0000 7f6b04723640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:21:45.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.050+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ae4005330 con 0x7f6afc06cac0 2024-03-29T11:21:45.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.051+0000 7f6aeeffd640 1 -- 172.21.15.19:0/1519946902 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f6af00520b0 con 0x7f6afc06cac0 2024-03-29T11:21:45.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.051+0000 7f6aeeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:45.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.051+0000 7f6aeeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:45.028 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:45.028 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 10 librados: watch_flush enter 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 10 librados: watch_flush exit 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 10 client.4332.objecter successfully canceled tick 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 >> v1:172.21.15.19:6816/3721232097 conn(0x7f6ae006e370 legacy=0x7f6ae0070830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 10 monclient: shutdown 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 >> v1:172.21.15.19:6789/0 conn(0x7f6afc06cac0 legacy=0x7f6afc1210a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 shutdown_connections 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 >> 172.21.15.19:0/1519946902 conn(0x7f6afc0675b0 msgr2=0x7f6afc143e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.054+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 shutdown_connections 2024-03-29T11:21:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.055+0000 7f6b04723640 1 -- 172.21.15.19:0/1519946902 wait complete. 2024-03-29T11:21:45.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:45.055+0000 7f6b04723640 1 librados: shutdown 2024-03-29T11:21:45.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:51.852 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:51.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.007+0000 7f3149093640 10 monclient: get_monmap_and_config 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.007+0000 7f3149093640 10 monclient: build_initial_monmap 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.007+0000 7f3149093640 10 monclient: monmap: 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:52.008223+0000 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:52.008223+0000 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:51.984 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:51.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.008+0000 7f3149093640 1 Processor -- start 2024-03-29T11:21:51.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.008+0000 7f3149093640 1 -- start start 2024-03-29T11:21:51.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.008+0000 7f3149093640 10 monclient: init 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient(hunting): picked mon.noname-a con 0x7f31441457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient(hunting): picked mon.noname-c con 0x7f314413f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient(hunting): picked mon.noname-b con 0x7f31440a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31440aaef0 con 0x7f31441457f0 2024-03-29T11:21:51.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3144145bd0 con 0x7f314413f650 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3144145eb0 con 0x7f31440a9d70 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f3149093640 10 monclient(hunting): authenticate will time out at 1032.272806s 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f314359e640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f31440a9d70 0x7f314414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54328/0 (socket says 172.21.15.19:54328) 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.009+0000 7f314359e640 1 -- 172.21.15.19:0/386948387 learned_addr learned my addr 172.21.15.19:0/386948387 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2838012027 0 0) 0x7f3144145bd0 con 0x7f314413f650 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312c003520 con 0x7f314413f650 2024-03-29T11:21:51.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145335038 0 0) 0x7f3144145eb0 con 0x7f31440a9d70 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.010+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3144145bd0 con 0x7f31440a9d70 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.011+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1367117327 0 0) 0x7f3144145bd0 con 0x7f31440a9d70 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.011+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 >> v1:172.21.15.62:6789/0 conn(0x7f314413f650 legacy=0x7f31440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:51.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.011+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 >> v1:172.21.15.19:6789/0 conn(0x7f31441457f0 legacy=0x7f314413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:51.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 1 monclient: found mon.noname-b 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3144146340 con 0x7f31440a9d70 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:51.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f313c0019f0 con 0x7f31440a9d70 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: dump: 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:51.991 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3149093640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3149093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f313c001b50 con 0x7f31440a9d70 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 1 -- 172.21.15.19:0/386948387 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f313c001f60 con 0x7f31440a9d70 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3141d9b640 10 monclient: dump: 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:51.992 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:51.993 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3149093640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.013+0000 7f3149093640 4 set_mon_vals no callback set 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.014+0000 7f3149093640 10 monclient: shutdown 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.014+0000 7f3149093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.014+0000 7f3149093640 1 -- 172.21.15.19:0/386948387 >> v1:172.21.15.19:6790/0 conn(0x7f31440a9d70 legacy=0x7f314414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.015+0000 7f3149093640 1 -- 172.21.15.19:0/386948387 shutdown_connections 2024-03-29T11:21:52.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.015+0000 7f3149093640 1 -- 172.21.15.19:0/386948387 >> 172.21.15.19:0/386948387 conn(0x7f3144066a00 msgr2=0x7f3144066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.015+0000 7f3149093640 1 -- 172.21.15.19:0/386948387 shutdown_connections 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.015+0000 7f3149093640 1 -- 172.21.15.19:0/386948387 wait complete. 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.016+0000 7f3149093640 10 monclient: build_initial_monmap 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.016+0000 7f3149093640 10 monclient: monmap: 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:52.002 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.016+0000 7f3149093640 1 librados: starting msgr at 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.016+0000 7f3149093640 1 librados: starting objecter 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 1 Processor -- start 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 1 -- start start 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 1 librados: setting wanted keys 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 1 librados: calling monclient init 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 10 monclient: init 2024-03-29T11:21:52.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 10 monclient(hunting): picked mon.noname-b con 0x7f31440a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f3149093640 10 monclient(hunting): picked mon.noname-c con 0x7f314413f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.017+0000 7f314259c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f314413f650 0x7f31441d43d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54586/0 (socket says 172.21.15.19:54586) 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f314259c640 1 -- 172.21.15.19:0/163058835 learned_addr learned my addr 172.21.15.19:0/163058835 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f312b7fe640 10 client.?.objecter ms_handle_connect 0x7f314413f650 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f312b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f3149093640 10 monclient(hunting): picked mon.noname-a con 0x7f31441457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3144141c80 con 0x7f31441457f0 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31441d7720 con 0x7f31440a9d70 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3144141ef0 con 0x7f314413f650 2024-03-29T11:21:52.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f3149093640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.018+0000 7f3149093640 10 monclient(hunting): authenticate will time out at 1032.281763s 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498882325 0 0) 0x7f3144141ef0 con 0x7f314413f650 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3130004eb0 con 0x7f314413f650 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4082766721 0 0) 0x7f3130004eb0 con 0x7f314413f650 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3144141ef0 con 0x7f314413f650 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3138001a10 con 0x7f314413f650 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:52.005 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 client.?.objecter ms_handle_connect 0x7f31440a9d70 2024-03-29T11:21:52.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.019+0000 7f312b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409892229 0 0) 0x7f31441d7720 con 0x7f31440a9d70 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3130004eb0 con 0x7f31440a9d70 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1839019892 0 0) 0x7f3144141ef0 con 0x7f314413f650 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 >> v1:172.21.15.62:6789/0 conn(0x7f31440a9d70 legacy=0x7f31441d7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.020+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 >> v1:172.21.15.19:6789/0 conn(0x7f31441457f0 legacy=0x7f3144142310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 1 monclient: found mon.c 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:52.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3144140230 con 0x7f314413f650 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 10 monclient: _renew_subs 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31441d4b40 con 0x7f314413f650 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 10 monclient: _renew_subs 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f314413ffd0 con 0x7f314413f650 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f3149093640 1 librados: init done 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3138001d10 con 0x7f314413f650 2024-03-29T11:21:52.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3138003a60 con 0x7f314413f650 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f314159a640 4 set_mon_vals no callback set 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.021+0000 7f312b7fe640 10 monclient: dump: 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:52.009 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.023+0000 7f3149093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.023+0000 7f3149093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.023+0000 7f3149093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.023+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f3138003bc0 con 0x7f314413f650 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.023+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3124005080 con 0x7f314413f650 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f3138084b90 con 0x7f314413f650 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f312b7fe640 10 client.4331.objecter ms_dispatch 0x7f3144000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f312b7fe640 3 client.4331.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f312b7fe640 3 client.4331.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f312b7fe640 20 client.4331.objecter dump_active .. 0 homeless 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f3142d9d640 10 monclient: get_auth_request con 0x7f313006de80 auth_method 0 2024-03-29T11:21:52.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.024+0000 7f312b7fe640 10 client.4331.objecter ms_handle_connect 0x7f313006de80 2024-03-29T11:21:52.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.033+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3138050470 con 0x7f314413f650 2024-03-29T11:21:52.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.033+0000 7f312b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:52.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.033+0000 7f312b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:52.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.359+0000 7f3149093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:52.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.359+0000 7f3149093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:52.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.359+0000 7f3149093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:21:52.336 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.359+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3124005330 con 0x7f314413f650 2024-03-29T11:21:52.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.360+0000 7f312b7fe640 1 -- 172.21.15.19:0/163058835 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f31380523a0 con 0x7f314413f650 2024-03-29T11:21:52.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.360+0000 7f312b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:52.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.360+0000 7f312b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:52.337 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:52.337 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:52.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 10 librados: watch_flush enter 2024-03-29T11:21:52.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 10 librados: watch_flush exit 2024-03-29T11:21:52.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 20 client.4331.objecter shutdown clearing up homeless session... 2024-03-29T11:21:52.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 10 client.4331.objecter successfully canceled tick 2024-03-29T11:21:52.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 >> v1:172.21.15.19:6816/3721232097 conn(0x7f313006de80 legacy=0x7f3130070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:52.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 10 monclient: shutdown 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.363+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 >> v1:172.21.15.19:6790/0 conn(0x7f314413f650 legacy=0x7f31441d43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.364+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 shutdown_connections 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.364+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 >> 172.21.15.19:0/163058835 conn(0x7f3144066a00 msgr2=0x7f31440b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.364+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 shutdown_connections 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.364+0000 7f3149093640 1 -- 172.21.15.19:0/163058835 wait complete. 2024-03-29T11:21:52.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:52.364+0000 7f3149093640 1 librados: shutdown 2024-03-29T11:21:52.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:59.161 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:59.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.301+0000 7f19d017b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:59.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.301+0000 7f19d017b640 10 monclient: build_initial_monmap 2024-03-29T11:21:59.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.301+0000 7f19d017b640 10 monclient: monmap: 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:21:59.302661+0000 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:21:59.302661+0000 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:59.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.302+0000 7f19d017b640 1 Processor -- start 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.303+0000 7f19d017b640 1 -- start start 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.303+0000 7f19d017b640 10 monclient: init 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient(hunting): picked mon.noname-b con 0x7f19c80ae250 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient(hunting): picked mon.noname-a con 0x7f19c813f970 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient(hunting): picked mon.noname-c con 0x7f19c8142800 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:59.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c80afe00 con 0x7f19c813f970 2024-03-29T11:21:59.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c8142c10 con 0x7f19c8142800 2024-03-29T11:21:59.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c80afb20 con 0x7f19c80ae250 2024-03-29T11:21:59.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:59.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19d017b640 10 monclient(hunting): authenticate will time out at 1039.567182s 2024-03-29T11:21:59.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19cdf17640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f19c80ae250 0x7f19c813f260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33756/0 (socket says 172.21.15.19:33756) 2024-03-29T11:21:59.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19cdf17640 1 -- 172.21.15.19:0/11200752 learned_addr learned my addr 172.21.15.19:0/11200752 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420204636 0 0) 0x7f19c80afb20 con 0x7f19c80ae250 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): _negotiate 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19b4003520 con 0x7f19c80ae250 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190795601 0 0) 0x7f19c8142c10 con 0x7f19c8142800 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): _negotiate 2024-03-29T11:21:59.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:59.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:59.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:21:59.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.304+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c80afb20 con 0x7f19c8142800 2024-03-29T11:21:59.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.305+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1123525667 0 0) 0x7f19c80afb20 con 0x7f19c8142800 2024-03-29T11:21:59.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.305+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 >> v1:172.21.15.19:6790/0 conn(0x7f19c80ae250 legacy=0x7f19c813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:59.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.305+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 >> v1:172.21.15.19:6789/0 conn(0x7f19c813f970 legacy=0x7f19c81420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: _finish_hunting 0 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 1 monclient: found mon.noname-c 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c80aab80 con 0x7f19c8142800 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: _finish_auth 0 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: _check_auth_tickets 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:59.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f19c00019f0 con 0x7f19c8142800 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: dump: 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:59.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19d017b640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:21:59.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19d017b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:59.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19c0001b50 con 0x7f19c8142800 2024-03-29T11:21:59.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:59.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19d017b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19d017b640 4 set_mon_vals no callback set 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 1 -- 172.21.15.19:0/11200752 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f19c0001f60 con 0x7f19c8142800 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.308+0000 7f19ccf15640 10 monclient: dump: 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:59.288 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.309+0000 7f19d017b640 10 monclient: shutdown 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.309+0000 7f19d017b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.309+0000 7f19d017b640 1 -- 172.21.15.19:0/11200752 >> v1:172.21.15.62:6789/0 conn(0x7f19c8142800 legacy=0x7f19c814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.309+0000 7f19d017b640 1 -- 172.21.15.19:0/11200752 shutdown_connections 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.309+0000 7f19d017b640 1 -- 172.21.15.19:0/11200752 >> 172.21.15.19:0/11200752 conn(0x7f19c80b35e0 msgr2=0x7f19c80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.310+0000 7f19d017b640 1 -- 172.21.15.19:0/11200752 shutdown_connections 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.310+0000 7f19d017b640 1 -- 172.21.15.19:0/11200752 wait complete. 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.310+0000 7f19d017b640 10 monclient: build_initial_monmap 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.310+0000 7f19d017b640 10 monclient: monmap: 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:21:59.289 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 1 librados: starting msgr at 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 1 librados: starting objecter 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 1 Processor -- start 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 1 -- start start 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 1 librados: setting wanted keys 2024-03-29T11:21:59.290 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 1 librados: calling monclient init 2024-03-29T11:21:59.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.311+0000 7f19d017b640 10 monclient: init 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient(hunting): picked mon.noname-a con 0x7f19c80ae250 addr v1:172.21.15.19:6789/0 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient(hunting): picked mon.noname-b con 0x7f19c813f970 addr v1:172.21.15.62:6789/0 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient(hunting): picked mon.noname-c con 0x7f19c8142800 addr v1:172.21.15.19:6790/0 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c8145e00 con 0x7f19c80ae250 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c81454b0 con 0x7f19c813f970 2024-03-29T11:21:59.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c8146030 con 0x7f19c8142800 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19d017b640 10 monclient(hunting): authenticate will time out at 1039.575685s 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19cd716640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f19c813f970 0x7f19c81382c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:45162/0 (socket says 172.21.15.19:45162) 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19cd716640 1 -- 172.21.15.19:0/1970262 learned_addr learned my addr 172.21.15.19:0/1970262 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.312+0000 7f19be7fc640 10 client.?.objecter ms_handle_connect 0x7f19c813f970 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263196483 0 0) 0x7f19c81454b0 con 0x7f19c813f970 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:21:59.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac004eb0 con 0x7f19c813f970 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908032219 0 0) 0x7f19ac004eb0 con 0x7f19c813f970 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19c81454b0 con 0x7f19c813f970 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f19b8001960 con 0x7f19c813f970 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.313+0000 7f19be7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:59.296 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.314+0000 7f19be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.314+0000 7f19be7fc640 10 client.?.objecter ms_handle_connect 0x7f19c8142800 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.314+0000 7f19be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.314+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 962612055 0 0) 0x7f19c81454b0 con 0x7f19c813f970 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.314+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 >> v1:172.21.15.19:6790/0 conn(0x7f19c8142800 legacy=0x7f19c81e6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.314+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 >> v1:172.21.15.19:6789/0 conn(0x7f19c80ae250 legacy=0x7f19c813ab30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 1 monclient: found mon.b 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:59.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c8146310 con 0x7f19c813f970 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 10 monclient: _renew_subs 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c8144f90 con 0x7f19c813f970 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 10 monclient: _renew_subs 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c81443e0 con 0x7f19c813f970 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19d017b640 1 librados: init done 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19b8001ac0 con 0x7f19c813f970 2024-03-29T11:21:59.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f19b8003a40 con 0x7f19c813f970 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19be7fc640 10 monclient: dump: 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:21:59.299 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:21:59.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.315+0000 7f19bffff640 4 set_mon_vals no callback set 2024-03-29T11:21:59.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.316+0000 7f19d017b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:59.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.318+0000 7f19d017b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:59.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.318+0000 7f19d017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:59.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.318+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b0005080 con 0x7f19c813f970 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f19b8003ba0 con 0x7f19c813f970 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f19b8084af0 con 0x7f19c813f970 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19be7fc640 10 client.4318.objecter ms_dispatch 0x7f19c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19be7fc640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19be7fc640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19be7fc640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.319+0000 7f19cdf17640 10 monclient: get_auth_request con 0x7f19ac06db60 auth_method 0 2024-03-29T11:21:59.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.320+0000 7f19be7fc640 10 client.4318.objecter ms_handle_connect 0x7f19ac06db60 2024-03-29T11:21:59.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.328+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19b80503d0 con 0x7f19c813f970 2024-03-29T11:21:59.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.328+0000 7f19be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:59.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.328+0000 7f19be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:59.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.653+0000 7f19d017b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:59.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.653+0000 7f19d017b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:59.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.653+0000 7f19d017b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:21:59.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.653+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19b0005330 con 0x7f19c813f970 2024-03-29T11:21:59.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.654+0000 7f19be7fc640 1 -- 172.21.15.19:0/1970262 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f19b8052300 con 0x7f19c813f970 2024-03-29T11:21:59.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.654+0000 7f19be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:59.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.654+0000 7f19be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:59.630 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:21:59.631 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:59.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.657+0000 7f19d017b640 10 librados: watch_flush enter 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 10 librados: watch_flush exit 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 10 client.4318.objecter successfully canceled tick 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 >> v1:172.21.15.19:6816/3721232097 conn(0x7f19ac06db60 legacy=0x7f19ac070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 10 monclient: shutdown 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 >> v1:172.21.15.62:6789/0 conn(0x7f19c813f970 legacy=0x7f19c81382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 shutdown_connections 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 >> 172.21.15.19:0/1970262 conn(0x7f19c80b35e0 msgr2=0x7f19c80b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 shutdown_connections 2024-03-29T11:21:59.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 -- 172.21.15.19:0/1970262 wait complete. 2024-03-29T11:21:59.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:21:59.658+0000 7f19d017b640 1 librados: shutdown 2024-03-29T11:21:59.648 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:06.454 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.636+0000 7ff77e172640 10 monclient: get_monmap_and_config 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.636+0000 7ff77e172640 10 monclient: build_initial_monmap 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.636+0000 7ff77e172640 10 monclient: monmap: 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:06.637612+0000 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:06.637612+0000 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:06.613 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:06.614 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:06.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.641+0000 7ff77e172640 1 Processor -- start 2024-03-29T11:22:06.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.641+0000 7ff77e172640 1 -- start start 2024-03-29T11:22:06.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.641+0000 7ff77e172640 10 monclient: init 2024-03-29T11:22:06.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:06.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:06.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient(hunting): picked mon.noname-c con 0x7ff7781422b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:06.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient(hunting): picked mon.noname-a con 0x7ff778142690 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:06.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient(hunting): picked mon.noname-b con 0x7ff7780b17d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:06.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff778142c80 con 0x7ff778142690 2024-03-29T11:22:06.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780aae40 con 0x7ff7781422b0 2024-03-29T11:22:06.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff77813fbd0 con 0x7ff7780b17d0 2024-03-29T11:22:06.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:06.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.642+0000 7ff77e172640 10 monclient(hunting): authenticate will time out at 1046.905534s 2024-03-29T11:22:06.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.645+0000 7ff777fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff7780b17d0 0x7ff7780b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48833/0 (socket says 172.21.15.19:48833) 2024-03-29T11:22:06.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.645+0000 7ff777fff640 1 -- 172.21.15.19:0/460801877 learned_addr learned my addr 172.21.15.19:0/460801877 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:06.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.647+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114379537 0 0) 0x7ff77813fbd0 con 0x7ff7780b17d0 2024-03-29T11:22:06.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.647+0000 7ff7767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:06.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.647+0000 7ff7767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:06.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.647+0000 7ff7767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:06.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.647+0000 7ff7767fc640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:22:06.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.647+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff760003520 con 0x7ff7780b17d0 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.648+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811720739 0 0) 0x7ff7780aae40 con 0x7ff7781422b0 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.648+0000 7ff7767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.648+0000 7ff7767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.648+0000 7ff7767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.648+0000 7ff7767fc640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.648+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff77813fbd0 con 0x7ff7781422b0 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.649+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877178893 0 0) 0x7ff760003520 con 0x7ff7780b17d0 2024-03-29T11:22:06.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.649+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 >> v1:172.21.15.62:6789/0 conn(0x7ff7781422b0 legacy=0x7ff77813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.649+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 >> v1:172.21.15.19:6789/0 conn(0x7ff778142690 legacy=0x7ff7780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff778151940 con 0x7ff7780b17d0 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff768001690 con 0x7ff7780b17d0 2024-03-29T11:22:06.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff77e172640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff77e172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.650+0000 7ff7767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff7767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff7767fc640 10 monclient: dump: 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:06.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff77e172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7680018b0 con 0x7ff7780b17d0 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff7767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff7767fc640 1 -- 172.21.15.19:0/460801877 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff768001bc0 con 0x7ff7780b17d0 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff77e172640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.651+0000 7ff77e172640 4 set_mon_vals no callback set 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.652+0000 7ff77e172640 10 monclient: shutdown 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.652+0000 7ff77e172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.652+0000 7ff77e172640 1 -- 172.21.15.19:0/460801877 >> v1:172.21.15.19:6790/0 conn(0x7ff7780b17d0 legacy=0x7ff7780b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.652+0000 7ff7767fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:22:06.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.652+0000 7ff77e172640 1 -- 172.21.15.19:0/460801877 shutdown_connections 2024-03-29T11:22:06.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.652+0000 7ff77e172640 1 -- 172.21.15.19:0/460801877 >> 172.21.15.19:0/460801877 conn(0x7ff778066a00 msgr2=0x7ff778066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:06.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.653+0000 7ff77e172640 1 -- 172.21.15.19:0/460801877 shutdown_connections 2024-03-29T11:22:06.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.655+0000 7ff77e172640 1 -- 172.21.15.19:0/460801877 wait complete. 2024-03-29T11:22:06.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.656+0000 7ff77e172640 10 monclient: build_initial_monmap 2024-03-29T11:22:06.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.656+0000 7ff77e172640 10 monclient: monmap: 2024-03-29T11:22:06.632 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:06.632 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:06.632 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:06.632 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:06.633 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:06.633 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:06.633 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:06.633 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:06.633 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:06.633 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.656+0000 7ff77e172640 1 librados: starting msgr at 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.656+0000 7ff77e172640 1 librados: starting objecter 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.656+0000 7ff77e172640 1 Processor -- start 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.657+0000 7ff77e172640 1 -- start start 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.657+0000 7ff77e172640 1 librados: setting wanted keys 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.657+0000 7ff77e172640 1 librados: calling monclient init 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.657+0000 7ff77e172640 10 monclient: init 2024-03-29T11:22:06.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient(hunting): picked mon.noname-a con 0x7ff7780b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient(hunting): picked mon.noname-c con 0x7ff7781422b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient(hunting): picked mon.noname-b con 0x7ff778142690 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780b1040 con 0x7ff7780b17d0 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780b12b0 con 0x7ff778142690 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7780afe20 con 0x7ff7781422b0 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:06.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.658+0000 7ff77e172640 10 monclient(hunting): authenticate will time out at 1046.921284s 2024-03-29T11:22:06.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.662+0000 7ff7777fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff7780b17d0 0x7ff7780af130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:57355/0 (socket says 172.21.15.19:57355) 2024-03-29T11:22:06.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.662+0000 7ff7777fe640 1 -- 172.21.15.19:0/3160763303 learned_addr learned my addr 172.21.15.19:0/3160763303 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.663+0000 7ff75ffff640 10 client.?.objecter ms_handle_connect 0x7ff7780b17d0 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440257081 0 0) 0x7ff7780b1040 con 0x7ff7780b17d0 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:22:06.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.664+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff764004eb0 con 0x7ff7780b17d0 2024-03-29T11:22:06.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 client.?.objecter ms_handle_connect 0x7ff7781422b0 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601811800 0 0) 0x7ff7780afe20 con 0x7ff7781422b0 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7780b1040 con 0x7ff7781422b0 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161967324 0 0) 0x7ff764004eb0 con 0x7ff7780b17d0 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7780afe20 con 0x7ff7780b17d0 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff76c001a10 con 0x7ff7780b17d0 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:06.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 478159434 0 0) 0x7ff7780b1040 con 0x7ff7781422b0 2024-03-29T11:22:06.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff764004eb0 con 0x7ff7781422b0 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff770001b80 con 0x7ff7781422b0 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient(hunting): dump: 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:06.647 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:06.648 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:06.648 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:06.648 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:06.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 741751891 0 0) 0x7ff7780afe20 con 0x7ff7780b17d0 2024-03-29T11:22:06.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 >> v1:172.21.15.19:6790/0 conn(0x7ff7781422b0 legacy=0x7ff7781dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 >> v1:172.21.15.62:6789/0 conn(0x7ff778142690 legacy=0x7ff7781e1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:06.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 monclient: found mon.a 2024-03-29T11:22:06.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:06.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:06.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7780affb0 con 0x7ff7780b17d0 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient: _finish_auth 0 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff75ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff77e172640 5 monclient: authenticate success, global_id 4338 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff77e172640 10 monclient: _renew_subs 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff77e172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.666+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7781db280 con 0x7ff7780b17d0 2024-03-29T11:22:06.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff77e172640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff77e172640 10 monclient: _renew_subs 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff77e172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7781db760 con 0x7ff7780b17d0 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff77e172640 1 librados: init done 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff76c001b70 con 0x7ff7780b17d0 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff75ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff76c003a60 con 0x7ff7780b17d0 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff75ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff75ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:06.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff75ffff640 10 monclient: dump: 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:06.652 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.667+0000 7ff7757fa640 4 set_mon_vals no callback set 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.668+0000 7ff77e172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.668+0000 7ff77e172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.668+0000 7ff77e172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.668+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff758005080 con 0x7ff7780b17d0 2024-03-29T11:22:06.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.669+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff76c003bc0 con 0x7ff7780b17d0 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff76c084b90 con 0x7ff7780b17d0 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff75ffff640 10 client.4338.objecter ms_dispatch 0x7ff778000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff75ffff640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff75ffff640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff75ffff640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff776ffd640 10 monclient: get_auth_request con 0x7ff76406dd50 auth_method 0 2024-03-29T11:22:06.654 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.670+0000 7ff75ffff640 10 client.4338.objecter ms_handle_connect 0x7ff76406dd50 2024-03-29T11:22:06.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.681+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff76c050470 con 0x7ff7780b17d0 2024-03-29T11:22:06.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.681+0000 7ff75ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:06.657 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:06.681+0000 7ff75ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:06.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.005+0000 7ff77e172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:06.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.005+0000 7ff77e172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:06.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.005+0000 7ff77e172640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:06.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.005+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff758005330 con 0x7ff7780b17d0 2024-03-29T11:22:06.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.007+0000 7ff75ffff640 1 -- 172.21.15.19:0/3160763303 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff76c0523a0 con 0x7ff7780b17d0 2024-03-29T11:22:06.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.007+0000 7ff75ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:06.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.007+0000 7ff75ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:06.983 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:06.984 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:06.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.010+0000 7ff77e172640 10 librados: watch_flush enter 2024-03-29T11:22:06.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 10 librados: watch_flush exit 2024-03-29T11:22:06.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-29T11:22:06.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 10 client.4338.objecter successfully canceled tick 2024-03-29T11:22:06.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff76406dd50 legacy=0x7ff764070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 10 monclient: shutdown 2024-03-29T11:22:06.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:06.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.011+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 >> v1:172.21.15.19:6789/0 conn(0x7ff7780b17d0 legacy=0x7ff7780af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:06.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.014+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 shutdown_connections 2024-03-29T11:22:06.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.014+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 >> 172.21.15.19:0/3160763303 conn(0x7ff778066a00 msgr2=0x7ff77813e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:06.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.015+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 shutdown_connections 2024-03-29T11:22:06.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.015+0000 7ff77e172640 1 -- 172.21.15.19:0/3160763303 wait complete. 2024-03-29T11:22:06.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:07.015+0000 7ff77e172640 1 librados: shutdown 2024-03-29T11:22:07.008 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:13.814 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:13.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.994+0000 7f55ab59e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:13.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.994+0000 7f55ab59e640 10 monclient: build_initial_monmap 2024-03-29T11:22:13.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.994+0000 7f55ab59e640 10 monclient: monmap: 2024-03-29T11:22:13.971 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:13.971 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:13.971 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:13.995577+0000 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:13.995577+0000 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:13.972 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:13.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.997+0000 7f55ab59e640 1 Processor -- start 2024-03-29T11:22:13.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.997+0000 7f55ab59e640 1 -- start start 2024-03-29T11:22:13.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.997+0000 7f55ab59e640 10 monclient: init 2024-03-29T11:22:13.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient(hunting): picked mon.noname-c con 0x7f55a4006890 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient(hunting): picked mon.noname-a con 0x7f55a4006c70 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient(hunting): picked mon.noname-b con 0x7f55a402b4c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a402c830 con 0x7f55a4006c70 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a402afa0 con 0x7f55a4006890 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a402aba0 con 0x7f55a402b4c0 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:13.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:13.999+0000 7f55ab59e640 10 monclient(hunting): authenticate will time out at 1054.262154s 2024-03-29T11:22:13.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.002+0000 7f55aa59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f55a4006890 0x7f55a40c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:55455/0 (socket says 172.21.15.19:55455) 2024-03-29T11:22:13.978 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.002+0000 7f55aa59c640 1 -- 172.21.15.19:0/2632211811 learned_addr learned my addr 172.21.15.19:0/2632211811 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:13.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.005+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370475497 0 0) 0x7f55a402aba0 con 0x7f55a402b4c0 2024-03-29T11:22:13.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.005+0000 7f55a959a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:13.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.005+0000 7f55a959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:13.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.005+0000 7f55a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:13.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.005+0000 7f55a959a640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:22:13.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.005+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5594003520 con 0x7f55a402b4c0 2024-03-29T11:22:13.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.006+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1985160176 0 0) 0x7f5594003520 con 0x7f55a402b4c0 2024-03-29T11:22:13.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.006+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 >> v1:172.21.15.62:6789/0 conn(0x7f55a4006890 legacy=0x7f55a40c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.006+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 >> v1:172.21.15.19:6789/0 conn(0x7f55a4006c70 legacy=0x7f55a40c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 1 monclient: found mon.noname-b 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a402ad30 con 0x7f55a402b4c0 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: _finish_auth 0 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f559c001690 con 0x7f55a402b4c0 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55a959a640 10 monclient: dump: 2024-03-29T11:22:13.985 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55ab59e640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.007+0000 7f55ab59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f559c001930 con 0x7f55a402b4c0 2024-03-29T11:22:13.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55a959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55a959a640 1 -- 172.21.15.19:0/2632211811 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f559c001cc0 con 0x7f55a402b4c0 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55a959a640 10 monclient: dump: 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:13.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55ab59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:13.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.008+0000 7f55ab59e640 4 set_mon_vals no callback set 2024-03-29T11:22:13.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.009+0000 7f55ab59e640 10 monclient: shutdown 2024-03-29T11:22:13.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.009+0000 7f55ab59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:13.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.009+0000 7f55ab59e640 1 -- 172.21.15.19:0/2632211811 >> v1:172.21.15.19:6790/0 conn(0x7f55a402b4c0 legacy=0x7f55a40cf970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:13.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.011+0000 7f55ab59e640 1 -- 172.21.15.19:0/2632211811 shutdown_connections 2024-03-29T11:22:13.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.011+0000 7f55ab59e640 1 -- 172.21.15.19:0/2632211811 >> 172.21.15.19:0/2632211811 conn(0x7f55a40058f0 msgr2=0x7f55a4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:13.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.013+0000 7f55ab59e640 1 -- 172.21.15.19:0/2632211811 shutdown_connections 2024-03-29T11:22:13.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.013+0000 7f55ab59e640 1 -- 172.21.15.19:0/2632211811 wait complete. 2024-03-29T11:22:13.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 10 monclient: build_initial_monmap 2024-03-29T11:22:13.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 10 monclient: monmap: 2024-03-29T11:22:13.990 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:13.990 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:13.990 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:13.990 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 1 librados: starting msgr at 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 1 librados: starting objecter 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 1 Processor -- start 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 1 -- start start 2024-03-29T11:22:13.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 1 librados: setting wanted keys 2024-03-29T11:22:13.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 1 librados: calling monclient init 2024-03-29T11:22:13.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.014+0000 7f55ab59e640 10 monclient: init 2024-03-29T11:22:13.993 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient(hunting): picked mon.noname-a con 0x7f55a4006890 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient(hunting): picked mon.noname-c con 0x7f55a402b4c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient(hunting): picked mon.noname-b con 0x7f55a40c15c0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a40c2620 con 0x7f55a4006890 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a40c2900 con 0x7f55a40c15c0 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a40c63a0 con 0x7f55a402b4c0 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:13.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.015+0000 7f55ab59e640 10 monclient(hunting): authenticate will time out at 1054.278738s 2024-03-29T11:22:13.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.022+0000 7f55aad9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f55a40c15c0 0x7f55a416bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:55577/0 (socket says 172.21.15.19:55577) 2024-03-29T11:22:13.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.022+0000 7f55aad9d640 1 -- 172.21.15.19:0/53722022 learned_addr learned my addr 172.21.15.19:0/53722022 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:13.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 10 client.?.objecter ms_handle_connect 0x7f55a40c15c0 2024-03-29T11:22:13.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:14.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416748414 0 0) 0x7f55a40c2900 con 0x7f55a40c15c0 2024-03-29T11:22:14.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.023+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f558c004eb0 con 0x7f55a40c15c0 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 429391890 0 0) 0x7f558c004eb0 con 0x7f55a40c15c0 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55a40c2900 con 0x7f55a40c15c0 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f559c001650 con 0x7f55a40c15c0 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 10 monclient(hunting): dump: 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:14.001 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:14.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.024+0000 7f559affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:14.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 709322083 0 0) 0x7f55a40c2900 con 0x7f55a40c15c0 2024-03-29T11:22:14.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 >> v1:172.21.15.19:6790/0 conn(0x7f55a402b4c0 legacy=0x7f55a40c2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:14.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 >> v1:172.21.15.19:6789/0 conn(0x7f55a4006890 legacy=0x7f55a40c6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:14.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:14.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 monclient: found mon.b 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a40c6030 con 0x7f55a40c15c0 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: _finish_auth 0 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 10 monclient: _renew_subs 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a40c34a0 con 0x7f55a40c15c0 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f559c003960 con 0x7f55a40c15c0 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:14.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f559c003db0 con 0x7f55a40c15c0 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559affd640 10 monclient: dump: 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:14.005 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f559bfff640 4 set_mon_vals no callback set 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 10 monclient: _renew_subs 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a4031cd0 con 0x7f55a40c15c0 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.025+0000 7f55ab59e640 1 librados: init done 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.027+0000 7f55ab59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.027+0000 7f55ab59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.027+0000 7f55ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.027+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a402a520 con 0x7f55a40c15c0 2024-03-29T11:22:14.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.028+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f559c003f10 con 0x7f55a40c15c0 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f559c084bc0 con 0x7f55a40c15c0 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f559affd640 10 client.4327.objecter ms_dispatch 0x7f55ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f559affd640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f559affd640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f559affd640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f55aa59c640 10 monclient: get_auth_request con 0x7f558c06dc90 auth_method 0 2024-03-29T11:22:14.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.029+0000 7f559affd640 10 client.4327.objecter ms_handle_connect 0x7f558c06dc90 2024-03-29T11:22:14.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.035+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f559c0504a0 con 0x7f55a40c15c0 2024-03-29T11:22:14.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.035+0000 7f559affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:14.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.035+0000 7f559affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:14.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.361+0000 7f55ab59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:14.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.361+0000 7f55ab59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:14.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.361+0000 7f55ab59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:14.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.361+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55a4004cd0 con 0x7f55a40c15c0 2024-03-29T11:22:14.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.364+0000 7f559affd640 1 -- 172.21.15.19:0/53722022 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f559c001210 con 0x7f55a40c15c0 2024-03-29T11:22:14.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.364+0000 7f559affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:14.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.364+0000 7f559affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:14.342 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:14.342 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 10 librados: watch_flush enter 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 10 librados: watch_flush exit 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 10 client.4327.objecter successfully canceled tick 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 >> v1:172.21.15.19:6816/3721232097 conn(0x7f558c06dc90 legacy=0x7f558c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 10 monclient: shutdown 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:14.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.369+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 >> v1:172.21.15.62:6789/0 conn(0x7f55a40c15c0 legacy=0x7f55a416bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:14.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.370+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 shutdown_connections 2024-03-29T11:22:14.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.370+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 >> 172.21.15.19:0/53722022 conn(0x7f55a40058f0 msgr2=0x7f55a4038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:14.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.370+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 shutdown_connections 2024-03-29T11:22:14.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.371+0000 7f55ab59e640 1 -- 172.21.15.19:0/53722022 wait complete. 2024-03-29T11:22:14.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:14.371+0000 7f55ab59e640 1 librados: shutdown 2024-03-29T11:22:14.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:21.172 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.353+0000 7fe9c5cf6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.353+0000 7fe9c5cf6640 10 monclient: build_initial_monmap 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.353+0000 7fe9c5cf6640 10 monclient: monmap: 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:21.354424+0000 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:21.354424+0000 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:21.331 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:21.332 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:21.332 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.358+0000 7fe9c5cf6640 1 Processor -- start 2024-03-29T11:22:21.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.358+0000 7fe9c5cf6640 1 -- start start 2024-03-29T11:22:21.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.358+0000 7fe9c5cf6640 10 monclient: init 2024-03-29T11:22:21.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:21.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:21.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c00a9930 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c0149dc0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c014ff40 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00aae60 con 0x7fe9c00a9930 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00b4310 con 0x7fe9c0149dc0 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00afdd0 con 0x7fe9c014ff40 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:21.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.359+0000 7fe9c5cf6640 10 monclient(hunting): authenticate will time out at 1061.622577s 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.363+0000 7fe9bffff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe9c014ff40 0x7fe9c0152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:44591/0 (socket says 172.21.15.19:44591) 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.363+0000 7fe9bffff640 1 -- 172.21.15.19:0/1272898573 learned_addr learned my addr 172.21.15.19:0/1272898573 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885030980 0 0) 0x7fe9c00afdd0 con 0x7fe9c014ff40 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:22:21.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a8003520 con 0x7fe9c014ff40 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896964752 0 0) 0x7fe9a8003520 con 0x7fe9c014ff40 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 >> v1:172.21.15.62:6789/0 conn(0x7fe9c0149dc0 legacy=0x7fe9c014a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 >> v1:172.21.15.19:6789/0 conn(0x7fe9c00a9930 legacy=0x7fe9c0143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 monclient: found mon.noname-b 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c014a950 con 0x7fe9c014ff40 2024-03-29T11:22:21.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:21.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:21.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:21.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe9b0001690 con 0x7fe9c014ff40 2024-03-29T11:22:21.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:21.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:21.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.364+0000 7fe9be7fc640 10 monclient: dump: 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:21.343 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b00019a0 con 0x7fe9c014ff40 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9be7fc640 1 -- 172.21.15.19:0/1272898573 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe9b0001d30 con 0x7fe9c014ff40 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9be7fc640 10 monclient: dump: 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:21.344 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.345 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:21.345 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:21.345 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:21.345 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9c5cf6640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9c5cf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.365+0000 7fe9c5cf6640 4 set_mon_vals no callback set 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.366+0000 7fe9c5cf6640 10 monclient: shutdown 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.366+0000 7fe9c5cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.366+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/1272898573 >> v1:172.21.15.19:6790/0 conn(0x7fe9c014ff40 legacy=0x7fe9c0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.366+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/1272898573 shutdown_connections 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.366+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/1272898573 >> 172.21.15.19:0/1272898573 conn(0x7fe9c00b35e0 msgr2=0x7fe9c00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.369+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/1272898573 shutdown_connections 2024-03-29T11:22:21.346 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.369+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/1272898573 wait complete. 2024-03-29T11:22:21.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.370+0000 7fe9c5cf6640 10 monclient: build_initial_monmap 2024-03-29T11:22:21.347 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.370+0000 7fe9c5cf6640 10 monclient: monmap: 2024-03-29T11:22:21.347 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:21.347 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.370+0000 7fe9c5cf6640 1 librados: starting msgr at 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.370+0000 7fe9c5cf6640 1 librados: starting objecter 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.370+0000 7fe9c5cf6640 1 Processor -- start 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 1 -- start start 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 1 librados: setting wanted keys 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 1 librados: calling monclient init 2024-03-29T11:22:21.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient: init 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c00a9930 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c0149dc0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c014ff40 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c006f090 con 0x7fe9c014ff40 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0142ee0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c006f2e0 con 0x7fe9c00a9930 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:21.351 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.371+0000 7fe9c5cf6640 10 monclient(hunting): authenticate will time out at 1061.634925s 2024-03-29T11:22:21.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.376+0000 7fe9bf7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe9c00a9930 0x7fe9c006fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:44665/0 (socket says 172.21.15.19:44665) 2024-03-29T11:22:21.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.376+0000 7fe9bf7fe640 1 -- 172.21.15.19:0/3950577688 learned_addr learned my addr 172.21.15.19:0/3950577688 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:21.353 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 client.?.objecter ms_handle_connect 0x7fe9c00a9930 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 client.?.objecter ms_handle_connect 0x7fe9c0149dc0 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3088932151 0 0) 0x7fe9c0142ee0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9ac004eb0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121191617 0 0) 0x7fe9c006f2e0 con 0x7fe9c00a9930 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:21.354 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:22:21.355 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0142ee0 con 0x7fe9c00a9930 2024-03-29T11:22:21.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2746355008 0 0) 0x7fe9ac004eb0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c006f2e0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe9b8001bc0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): dump: 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:21.358 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475106534 0 0) 0x7fe9c0142ee0 con 0x7fe9c00a9930 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9ac004eb0 con 0x7fe9c00a9930 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe9b40018a0 con 0x7fe9c00a9930 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 10 monclient(hunting): dump: 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.359 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2952706235 0 0) 0x7fe9c006f2e0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 >> v1:172.21.15.19:6790/0 conn(0x7fe9c00a9930 legacy=0x7fe9c006fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.378+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 >> v1:172.21.15.19:6789/0 conn(0x7fe9c014ff40 legacy=0x7fe9c01e6820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 1 monclient: found mon.b 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c013e040 con 0x7fe9c0149dc0 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.379+0000 7fe9a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 5 monclient: authenticate success, global_id 4330 2024-03-29T11:22:21.360 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 10 monclient: _renew_subs 2024-03-29T11:22:21.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:21.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c013f350 con 0x7fe9c0149dc0 2024-03-29T11:22:21.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:21.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 10 monclient: _renew_subs 2024-03-29T11:22:21.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:21.361 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c0142800 con 0x7fe9c0149dc0 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.380+0000 7fe9c5cf6640 1 librados: init done 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.381+0000 7fe9c5cf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9c5cf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9c5cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0075f10 con 0x7fe9c0149dc0 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b80020d0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe9b8003b50 con 0x7fe9c0149dc0 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9a7fff640 10 monclient: dump: 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.362 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.382+0000 7fe9bdffb640 4 set_mon_vals no callback set 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.384+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fe9b8019be0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fe9b8083ab0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9a7fff640 10 client.4330.objecter ms_dispatch 0x7fe9c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9a7fff640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9a7fff640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9a7fff640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-29T11:22:21.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9bf7fe640 10 monclient: get_auth_request con 0x7fe9ac06dfe0 auth_method 0 2024-03-29T11:22:21.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.385+0000 7fe9a7fff640 10 client.4330.objecter ms_handle_connect 0x7fe9ac06dfe0 2024-03-29T11:22:21.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.391+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9b804f1c0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.391+0000 7fe9a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:21.367 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.391+0000 7fe9a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.724+0000 7fe9c5cf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.724+0000 7fe9c5cf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.724+0000 7fe9c5cf6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:21.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.725+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c006ab70 con 0x7fe9c0149dc0 2024-03-29T11:22:21.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.726+0000 7fe9a7fff640 1 -- 172.21.15.19:0/3950577688 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fe9b80512c0 con 0x7fe9c0149dc0 2024-03-29T11:22:21.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.726+0000 7fe9a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:21.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.726+0000 7fe9a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:21.702 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:21.703 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.729+0000 7fe9c5cf6640 10 librados: watch_flush enter 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.729+0000 7fe9c5cf6640 10 librados: watch_flush exit 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.729+0000 7fe9c5cf6640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.729+0000 7fe9c5cf6640 10 client.4330.objecter successfully canceled tick 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.729+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 >> v1:172.21.15.19:6816/3721232097 conn(0x7fe9ac06dfe0 legacy=0x7fe9ac0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.730+0000 7fe9c5cf6640 10 monclient: shutdown 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.730+0000 7fe9c5cf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:21.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.730+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 >> v1:172.21.15.62:6789/0 conn(0x7fe9c0149dc0 legacy=0x7fe9c01e30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:21.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.732+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 shutdown_connections 2024-03-29T11:22:21.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.732+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 >> 172.21.15.19:0/3950577688 conn(0x7fe9c00b35e0 msgr2=0x7fe9c00b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:21.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.733+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 shutdown_connections 2024-03-29T11:22:21.710 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.734+0000 7fe9c5cf6640 1 -- 172.21.15.19:0/3950577688 wait complete. 2024-03-29T11:22:21.713 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:21.737+0000 7fe9c5cf6640 1 librados: shutdown 2024-03-29T11:22:21.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:28.539 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:28.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.711+0000 7f3828057640 10 monclient: get_monmap_and_config 2024-03-29T11:22:28.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.711+0000 7f3828057640 10 monclient: build_initial_monmap 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.711+0000 7f3828057640 10 monclient: monmap: 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:28.712291+0000 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:28.712291+0000 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.712+0000 7f3828057640 1 Processor -- start 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.712+0000 7f3828057640 1 -- start start 2024-03-29T11:22:28.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.712+0000 7f3828057640 10 monclient: init 2024-03-29T11:22:28.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:28.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:28.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient(hunting): picked mon.noname-a con 0x7f38200a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:28.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient(hunting): picked mon.noname-b con 0x7f382013e1e0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:28.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient(hunting): picked mon.noname-c con 0x7f3820071c30 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200ab300 con 0x7f38200a9750 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200ac2f0 con 0x7f3820071c30 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200ad8d0 con 0x7f382013e1e0 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.713+0000 7f3828057640 10 monclient(hunting): authenticate will time out at 1068.976823s 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f38255f2640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f382013e1e0 0x7f38200ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:38053/0 (socket says 172.21.15.19:38053) 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f38255f2640 1 -- 172.21.15.19:0/2493125996 learned_addr learned my addr 172.21.15.19:0/2493125996 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083842404 0 0) 0x7f38200ad8d0 con 0x7f382013e1e0 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f3824df1640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f3824df1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f3824df1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f3824df1640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.717+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c003520 con 0x7f382013e1e0 2024-03-29T11:22:28.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077441401 0 0) 0x7f38200ac2f0 con 0x7f3820071c30 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38200ad8d0 con 0x7f3820071c30 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3457364 0 0) 0x7f380c003520 con 0x7f382013e1e0 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 >> v1:172.21.15.62:6789/0 conn(0x7f3820071c30 legacy=0x7f38200ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.718+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 >> v1:172.21.15.19:6789/0 conn(0x7f38200a9750 legacy=0x7f382013dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: _finish_hunting 0 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 1 monclient: found mon.noname-b 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38200ada60 con 0x7f382013e1e0 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: _finish_auth 0 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: _check_auth_tickets 2024-03-29T11:22:28.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f381c001610 con 0x7f382013e1e0 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: dump: 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.698 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3828057640 5 monclient: authenticate success, global_id 4355 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3828057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f381c001770 con 0x7f382013e1e0 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 1 -- 172.21.15.19:0/2493125996 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f381c001bc0 con 0x7f382013e1e0 2024-03-29T11:22:28.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.719+0000 7f3824df1640 10 monclient: dump: 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:28.700 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.721+0000 7f3828057640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.721+0000 7f3828057640 4 set_mon_vals no callback set 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.722+0000 7f3828057640 10 monclient: shutdown 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.722+0000 7f3828057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.722+0000 7f3828057640 1 -- 172.21.15.19:0/2493125996 >> v1:172.21.15.19:6790/0 conn(0x7f382013e1e0 legacy=0x7f38200ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.722+0000 7f3828057640 1 -- 172.21.15.19:0/2493125996 shutdown_connections 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.722+0000 7f3828057640 1 -- 172.21.15.19:0/2493125996 >> 172.21.15.19:0/2493125996 conn(0x7f382006b660 msgr2=0x7f382006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:28.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.723+0000 7f3828057640 1 -- 172.21.15.19:0/2493125996 shutdown_connections 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.723+0000 7f3828057640 1 -- 172.21.15.19:0/2493125996 wait complete. 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 10 monclient: build_initial_monmap 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 10 monclient: monmap: 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:28.704 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 1 librados: starting msgr at 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 1 librados: starting objecter 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 1 Processor -- start 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 1 -- start start 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 1 librados: setting wanted keys 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 1 librados: calling monclient init 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.724+0000 7f3828057640 10 monclient: init 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:28.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient(hunting): picked mon.noname-a con 0x7f3820071c30 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient(hunting): picked mon.noname-b con 0x7f38200a9750 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient(hunting): picked mon.noname-c con 0x7f382013a140 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200b2630 con 0x7f3820071c30 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006f530 con 0x7f38200a9750 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006ee30 con 0x7f382013a140 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:28.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.725+0000 7f3828057640 10 monclient(hunting): authenticate will time out at 1068.988735s 2024-03-29T11:22:28.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.733+0000 7f38265f4640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f382013a140 0x7f38201e65f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:38155/0 (socket says 172.21.15.19:38155) 2024-03-29T11:22:28.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.733+0000 7f38265f4640 1 -- 172.21.15.19:0/897378212 learned_addr learned my addr 172.21.15.19:0/897378212 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:28.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.733+0000 7f38167fc640 10 client.?.objecter ms_handle_connect 0x7f382013a140 2024-03-29T11:22:28.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:28.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 client.?.objecter ms_handle_connect 0x7f38200a9750 2024-03-29T11:22:28.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:28.714 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512735100 0 0) 0x7f382006f530 con 0x7f38200a9750 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804004eb0 con 0x7f38200a9750 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789256087 0 0) 0x7f382006ee30 con 0x7f382013a140 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.734+0000 7f38167fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382006f530 con 0x7f382013a140 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491960091 0 0) 0x7f3804004eb0 con 0x7f38200a9750 2024-03-29T11:22:28.715 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f382006ee30 con 0x7f38200a9750 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f381c001a80 con 0x7f38200a9750 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 10 monclient(hunting): dump: 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:28.716 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1243248201 0 0) 0x7f382006ee30 con 0x7f38200a9750 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 >> v1:172.21.15.19:6790/0 conn(0x7f382013a140 legacy=0x7f38201e65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.735+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 >> v1:172.21.15.19:6789/0 conn(0x7f3820071c30 legacy=0x7f382006fa10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 1 monclient: found mon.b 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38200b3150 con 0x7f38200a9750 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f38167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:28.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 10 monclient: _renew_subs 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f382006dee0 con 0x7f38200a9750 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 10 monclient: _renew_subs 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38201e7ef0 con 0x7f38200a9750 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.736+0000 7f3828057640 1 librados: init done 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.737+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f381c004a90 con 0x7f38200a9750 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.737+0000 7f38167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.737+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f381c004da0 con 0x7f38200a9750 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.737+0000 7f38167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.737+0000 7f38167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:28.718 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.737+0000 7f38167fc640 10 monclient: dump: 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:28.719 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:28.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.738+0000 7f3817fff640 4 set_mon_vals no callback set 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.739+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f381c01ad70 con 0x7f38200a9750 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.739+0000 7f3828057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.739+0000 7f3828057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.739+0000 7f3828057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.739+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3808005080 con 0x7f38200a9750 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.740+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f381c084a20 con 0x7f38200a9750 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.740+0000 7f38167fc640 10 client.4336.objecter ms_dispatch 0x7f3820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.740+0000 7f38167fc640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.740+0000 7f38167fc640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.740+0000 7f38167fc640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.740+0000 7f3825df3640 10 monclient: get_auth_request con 0x7f380406dce0 auth_method 0 2024-03-29T11:22:28.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.742+0000 7f38167fc640 10 client.4336.objecter ms_handle_connect 0x7f380406dce0 2024-03-29T11:22:28.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.748+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f381c050300 con 0x7f38200a9750 2024-03-29T11:22:28.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.748+0000 7f38167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:28.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:28.748+0000 7f38167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:29.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.089+0000 7f3828057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:29.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.089+0000 7f3828057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:29.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.089+0000 7f3828057640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:29.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.089+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3808005330 con 0x7f38200a9750 2024-03-29T11:22:29.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.093+0000 7f38167fc640 1 -- 172.21.15.19:0/897378212 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f381c001210 con 0x7f38200a9750 2024-03-29T11:22:29.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.093+0000 7f38167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:29.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.093+0000 7f38167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:29.070 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:29.070 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 10 librados: watch_flush enter 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 10 librados: watch_flush exit 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 >> v1:172.21.15.19:6816/3721232097 conn(0x7f380406dce0 legacy=0x7f38040701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 10 monclient: shutdown 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:29.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 >> v1:172.21.15.62:6789/0 conn(0x7f38200a9750 legacy=0x7f3820070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:29.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 shutdown_connections 2024-03-29T11:22:29.074 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.097+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 >> 172.21.15.19:0/897378212 conn(0x7f382006b660 msgr2=0x7f382013bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.099+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 shutdown_connections 2024-03-29T11:22:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.099+0000 7f3828057640 1 -- 172.21.15.19:0/897378212 wait complete. 2024-03-29T11:22:29.075 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:29.099+0000 7f3828057640 1 librados: shutdown 2024-03-29T11:22:29.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:35.898 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:36.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.072+0000 7f3acae38640 10 monclient: get_monmap_and_config 2024-03-29T11:22:36.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.072+0000 7f3acae38640 10 monclient: build_initial_monmap 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.072+0000 7f3acae38640 10 monclient: monmap: 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:36.073146+0000 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:36.073146+0000 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:36.049 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.073+0000 7f3acae38640 1 Processor -- start 2024-03-29T11:22:36.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.074+0000 7f3acae38640 1 -- start start 2024-03-29T11:22:36.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.074+0000 7f3acae38640 10 monclient: init 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient(hunting): picked mon.noname-b con 0x7f3abc006ad0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient(hunting): picked mon.noname-a con 0x7f3abc0c5120 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient(hunting): picked mon.noname-c con 0x7f3abc0cb780 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc008ed0 con 0x7f3abc0c5120 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc02ffb0 con 0x7f3abc0cb780 2024-03-29T11:22:36.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc0cbc10 con 0x7f3abc006ad0 2024-03-29T11:22:36.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:36.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.075+0000 7f3acae38640 10 monclient(hunting): authenticate will time out at 1076.338941s 2024-03-29T11:22:36.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.076+0000 7f3ac9e36640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f3abc006ad0 0x7f3abc0c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:37136/0 (socket says 172.21.15.19:37136) 2024-03-29T11:22:36.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.076+0000 7f3ac9e36640 1 -- 172.21.15.19:0/1672894131 learned_addr learned my addr 172.21.15.19:0/1672894131 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:36.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781106000 0 0) 0x7f3abc02ffb0 con 0x7f3abc0cb780 2024-03-29T11:22:36.053 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _negotiate 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab0003520 con 0x7f3abc0cb780 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935471883 0 0) 0x7f3abc0cbc10 con 0x7f3abc006ad0 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _negotiate 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab0003840 con 0x7f3abc006ad0 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591915553 0 0) 0x7f3abc008ed0 con 0x7f3abc0c5120 2024-03-29T11:22:36.054 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _negotiate 2024-03-29T11:22:36.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:36.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:36.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:22:36.055 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc0cbc10 con 0x7f3abc0c5120 2024-03-29T11:22:36.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 362613583 0 0) 0x7f3ab0003520 con 0x7f3abc0cb780 2024-03-29T11:22:36.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.077+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 >> v1:172.21.15.19:6790/0 conn(0x7f3abc006ad0 legacy=0x7f3abc0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 >> v1:172.21.15.19:6789/0 conn(0x7f3abc0c5120 legacy=0x7f3abc0cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: _finish_hunting 0 2024-03-29T11:22:36.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 1 monclient: found mon.noname-c 2024-03-29T11:22:36.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc030220 con 0x7f3abc0cb780 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: _finish_auth 0 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: _check_auth_tickets 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3ac0001670 con 0x7f3abc0cb780 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: dump: 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.057 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3acae38640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3acae38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ac0001850 con 0x7f3abc0cb780 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 1 -- 172.21.15.19:0/1672894131 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3ac0001670 con 0x7f3abc0cb780 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3ac8e34640 10 monclient: dump: 2024-03-29T11:22:36.058 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3acae38640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.078+0000 7f3acae38640 4 set_mon_vals no callback set 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.079+0000 7f3acae38640 10 monclient: shutdown 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.079+0000 7f3acae38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:36.059 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.079+0000 7f3acae38640 1 -- 172.21.15.19:0/1672894131 >> v1:172.21.15.62:6789/0 conn(0x7f3abc0cb780 legacy=0x7f3abc0d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.079+0000 7f3acae38640 1 -- 172.21.15.19:0/1672894131 shutdown_connections 2024-03-29T11:22:36.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.079+0000 7f3acae38640 1 -- 172.21.15.19:0/1672894131 >> 172.21.15.19:0/1672894131 conn(0x7f3abc0058f0 msgr2=0x7f3abc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:36.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.080+0000 7f3acae38640 1 -- 172.21.15.19:0/1672894131 shutdown_connections 2024-03-29T11:22:36.060 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.080+0000 7f3acae38640 1 -- 172.21.15.19:0/1672894131 wait complete. 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 10 monclient: build_initial_monmap 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 10 monclient: monmap: 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:36.063 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 1 librados: starting msgr at 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 1 librados: starting objecter 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 1 Processor -- start 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 1 -- start start 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 1 librados: setting wanted keys 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 1 librados: calling monclient init 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.081+0000 7f3acae38640 10 monclient: init 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient(hunting): picked mon.noname-a con 0x7f3abc006ad0 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient(hunting): picked mon.noname-b con 0x7f3abc0c5120 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient(hunting): picked mon.noname-c con 0x7f3abc0cb780 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:36.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc0bdeb0 con 0x7f3abc006ad0 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc03ab10 con 0x7f3abc0c5120 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3abc0b2dc0 con 0x7f3abc0cb780 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3acae38640 10 monclient(hunting): authenticate will time out at 1076.345632s 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3ac9e36640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f3abc006ad0 0x7f3abc036710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42904/0 (socket says 172.21.15.19:42904) 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3ac9e36640 1 -- 172.21.15.19:0/3405091225 learned_addr learned my addr 172.21.15.19:0/3405091225 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3aba7fc640 10 client.?.objecter ms_handle_connect 0x7f3abc006ad0 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.082+0000 7f3aba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 client.?.objecter ms_handle_connect 0x7f3abc0c5120 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671551489 0 0) 0x7f3abc0bdeb0 con 0x7f3abc006ad0 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:36.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa8004eb0 con 0x7f3abc006ad0 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815822754 0 0) 0x7f3abc03ab10 con 0x7f3abc0c5120 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3abc0bdeb0 con 0x7f3abc0c5120 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 client.?.objecter ms_handle_connect 0x7f3abc0cb780 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 848104629 0 0) 0x7f3aa8004eb0 con 0x7f3abc006ad0 2024-03-29T11:22:36.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3abc03ab10 con 0x7f3abc006ad0 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.083+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3ab4001930 con 0x7f3abc006ad0 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:36.067 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1403034820 0 0) 0x7f3abc0bdeb0 con 0x7f3abc0c5120 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3aa8004eb0 con 0x7f3abc0c5120 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3ac404fdd0 con 0x7f3abc0c5120 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.068 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1648897654 0 0) 0x7f3abc03ab10 con 0x7f3abc006ad0 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 >> v1:172.21.15.19:6790/0 conn(0x7f3abc0cb780 legacy=0x7f3abc1679a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 >> v1:172.21.15.62:6789/0 conn(0x7f3abc0c5120 legacy=0x7f3abc038ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 monclient: found mon.a 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc037590 con 0x7f3abc006ad0 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:36.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 5 monclient: authenticate success, global_id 4347 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 10 monclient: _renew_subs 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc037190 con 0x7f3abc006ad0 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 10 monclient: _renew_subs 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc0397b0 con 0x7f3abc006ad0 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3acae38640 1 librados: init done 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ab4001a90 con 0x7f3abc006ad0 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3aba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.084+0000 7f3abb7fe640 4 set_mon_vals no callback set 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.085+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f3ab4004c30 con 0x7f3abc006ad0 2024-03-29T11:22:36.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.085+0000 7f3aba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.085+0000 7f3aba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.085+0000 7f3aba7fc640 10 monclient: dump: 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.086+0000 7f3acae38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.086+0000 7f3acae38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:36.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.086+0000 7f3acae38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:36.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.086+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac005080 con 0x7f3abc006ad0 2024-03-29T11:22:36.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.088+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f3ab401ac00 con 0x7f3abc006ad0 2024-03-29T11:22:36.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.089+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f3ab4084b20 con 0x7f3abc006ad0 2024-03-29T11:22:36.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.089+0000 7f3aba7fc640 10 client.4347.objecter ms_dispatch 0x7f3ac4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:36.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.089+0000 7f3aba7fc640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:36.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.089+0000 7f3aba7fc640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:36.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.089+0000 7f3aba7fc640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-29T11:22:36.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.089+0000 7f3ac9635640 10 monclient: get_auth_request con 0x7f3aa806dc70 auth_method 0 2024-03-29T11:22:36.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.090+0000 7f3aba7fc640 10 client.4347.objecter ms_handle_connect 0x7f3aa806dc70 2024-03-29T11:22:36.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.101+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ab4050400 con 0x7f3abc006ad0 2024-03-29T11:22:36.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.102+0000 7f3aba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:36.078 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.102+0000 7f3aba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:36.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.470+0000 7f3acae38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:36.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.470+0000 7f3acae38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:36.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.470+0000 7f3acae38640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:22:36.447 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.470+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aac005600 con 0x7f3abc006ad0 2024-03-29T11:22:36.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.471+0000 7f3aba7fc640 1 -- 172.21.15.19:0/3405091225 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f3ab4052330 con 0x7f3abc006ad0 2024-03-29T11:22:36.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.471+0000 7f3aba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:36.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.471+0000 7f3aba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:36.448 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:36.448 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:36.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 10 librados: watch_flush enter 2024-03-29T11:22:36.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 10 librados: watch_flush exit 2024-03-29T11:22:36.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-29T11:22:36.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 10 client.4347.objecter successfully canceled tick 2024-03-29T11:22:36.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 >> v1:172.21.15.19:6816/3721232097 conn(0x7f3aa806dc70 legacy=0x7f3aa8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 10 monclient: shutdown 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 >> v1:172.21.15.19:6789/0 conn(0x7f3abc006ad0 legacy=0x7f3abc036710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 shutdown_connections 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 >> 172.21.15.19:0/3405091225 conn(0x7f3abc0058f0 msgr2=0x7f3abc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.475+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 shutdown_connections 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.476+0000 7f3acae38640 1 -- 172.21.15.19:0/3405091225 wait complete. 2024-03-29T11:22:36.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:36.476+0000 7f3acae38640 1 librados: shutdown 2024-03-29T11:22:36.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:43.271 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:43.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.457+0000 7f757590e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:43.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.457+0000 7f757590e640 10 monclient: build_initial_monmap 2024-03-29T11:22:43.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.457+0000 7f757590e640 10 monclient: monmap: 2024-03-29T11:22:43.435 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:43.435 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:43.458488+0000 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:43.458488+0000 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.459+0000 7f757590e640 1 Processor -- start 2024-03-29T11:22:43.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.459+0000 7f757590e640 1 -- start start 2024-03-29T11:22:43.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.459+0000 7f757590e640 10 monclient: init 2024-03-29T11:22:43.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.460+0000 7f757590e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:43.437 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.460+0000 7f757590e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 10 monclient(hunting): picked mon.noname-c con 0x7f75701422b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 10 monclient(hunting): picked mon.noname-b con 0x7f7570142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 10 monclient(hunting): picked mon.noname-a con 0x7f75700b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7570142c80 con 0x7f75700b17d0 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75700aae40 con 0x7f75701422b0 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757013fbd0 con 0x7f7570142690 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f757590e640 10 monclient(hunting): authenticate will time out at 1083.724109s 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756f7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f75700b17d0 0x7f75700b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:52814/0 (socket says 172.21.15.19:52814) 2024-03-29T11:22:43.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756f7fe640 1 -- 172.21.15.19:0/1528715155 learned_addr learned my addr 172.21.15.19:0/1528715155 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:43.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641466200 0 0) 0x7f75700aae40 con 0x7f75701422b0 2024-03-29T11:22:43.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:43.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:43.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:43.440 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.461+0000 7f756dffb640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7558003520 con 0x7f75701422b0 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105362100 0 0) 0x7f757013fbd0 con 0x7f7570142690 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75700aae40 con 0x7f7570142690 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437409055 0 0) 0x7f7570142c80 con 0x7f75700b17d0 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:22:43.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757013fbd0 con 0x7f75700b17d0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2005859109 0 0) 0x7f7558003520 con 0x7f75701422b0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 >> v1:172.21.15.19:6790/0 conn(0x7f7570142690 legacy=0x7f75700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 >> v1:172.21.15.19:6789/0 conn(0x7f75700b17d0 legacy=0x7f75700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 monclient: found mon.noname-c 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7570151940 con 0x7f75701422b0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7568001a10 con 0x7f75701422b0 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:43.442 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.462+0000 7f756dffb640 10 monclient: dump: 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f757590e640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f757590e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7568001b70 con 0x7f75701422b0 2024-03-29T11:22:43.443 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f756dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f756dffb640 1 -- 172.21.15.19:0/1528715155 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7568001f00 con 0x7f75701422b0 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f756dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f756dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f756dffb640 10 monclient: dump: 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:43.444 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f757590e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:43.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.463+0000 7f757590e640 4 set_mon_vals no callback set 2024-03-29T11:22:43.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.464+0000 7f757590e640 10 monclient: shutdown 2024-03-29T11:22:43.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.464+0000 7f757590e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:43.448 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.464+0000 7f757590e640 1 -- 172.21.15.19:0/1528715155 >> v1:172.21.15.62:6789/0 conn(0x7f75701422b0 legacy=0x7f757013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.464+0000 7f757590e640 1 -- 172.21.15.19:0/1528715155 shutdown_connections 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.464+0000 7f757590e640 1 -- 172.21.15.19:0/1528715155 >> 172.21.15.19:0/1528715155 conn(0x7f7570066a00 msgr2=0x7f7570066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.465+0000 7f757590e640 1 -- 172.21.15.19:0/1528715155 shutdown_connections 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.465+0000 7f757590e640 1 -- 172.21.15.19:0/1528715155 wait complete. 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.466+0000 7f757590e640 10 monclient: build_initial_monmap 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.466+0000 7f757590e640 10 monclient: monmap: 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:43.449 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.466+0000 7f757590e640 1 librados: starting msgr at 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.466+0000 7f757590e640 1 librados: starting objecter 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.466+0000 7f757590e640 1 Processor -- start 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.467+0000 7f757590e640 1 -- start start 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.467+0000 7f757590e640 1 librados: setting wanted keys 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.467+0000 7f757590e640 1 librados: calling monclient init 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.467+0000 7f757590e640 10 monclient: init 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient(hunting): picked mon.noname-b con 0x7f75700b17d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:43.450 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient(hunting): picked mon.noname-a con 0x7f75701422b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient(hunting): picked mon.noname-c con 0x7f7570142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757006f8f0 con 0x7f75701422b0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757006fb60 con 0x7f75700b17d0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757006fe40 con 0x7f7570142690 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f757590e640 10 monclient(hunting): authenticate will time out at 1083.731175s 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f756effd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f75700b17d0 0x7f757006de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:57788/0 (socket says 172.21.15.19:57788) 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f756effd640 1 -- 172.21.15.19:0/891927209 learned_addr learned my addr 172.21.15.19:0/891927209 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f7557fff640 10 client.?.objecter ms_handle_connect 0x7f75700b17d0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f7557fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f7557fff640 10 client.?.objecter ms_handle_connect 0x7f75701422b0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.468+0000 7f7557fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400357640 0 0) 0x7f757006fb60 con 0x7f75700b17d0 2024-03-29T11:22:43.451 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c004eb0 con 0x7f75700b17d0 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487712471 0 0) 0x7f757006f8f0 con 0x7f75701422b0 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f757006fb60 con 0x7f75701422b0 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 client.?.objecter ms_handle_connect 0x7f7570142690 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.469+0000 7f7557fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4202812259 0 0) 0x7f755c004eb0 con 0x7f75700b17d0 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f757006f8f0 con 0x7f75700b17d0 2024-03-29T11:22:43.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7568001e80 con 0x7f75700b17d0 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): dump: 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:43.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199925696 0 0) 0x7f757006fe40 con 0x7f7570142690 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f755c004eb0 con 0x7f7570142690 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 207605492 0 0) 0x7f757006fb60 con 0x7f75701422b0 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f757006fe40 con 0x7f75701422b0 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7560001850 con 0x7f75701422b0 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 10 monclient(hunting): dump: 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:43.454 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4269785649 0 0) 0x7f757006f8f0 con 0x7f75700b17d0 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.470+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 >> v1:172.21.15.19:6790/0 conn(0x7f7570142690 legacy=0x7f75701e7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 >> v1:172.21.15.19:6789/0 conn(0x7f75701422b0 legacy=0x7f757013bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 1 monclient: found mon.b 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:43.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757014e530 con 0x7f75700b17d0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 10 monclient: _renew_subs 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75700712f0 con 0x7f75700b17d0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 10 monclient: _renew_subs 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f757006e5a0 con 0x7f75700b17d0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f757590e640 1 librados: init done 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7568003c20 con 0x7f75700b17d0 2024-03-29T11:22:43.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7568003ff0 con 0x7f75700b17d0 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f756cff9640 4 set_mon_vals no callback set 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.471+0000 7f7557fff640 10 monclient: dump: 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:43.457 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:43.458 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:43.458 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.473+0000 7f757590e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.473+0000 7f757590e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.473+0000 7f757590e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.473+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7550005080 con 0x7f75700b17d0 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.474+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f7568004150 con 0x7f75700b17d0 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f756e7fc640 10 monclient: get_auth_request con 0x7f755c06e050 auth_method 0 2024-03-29T11:22:43.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f7568084e90 con 0x7f75700b17d0 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f7557fff640 10 client.4348.objecter ms_dispatch 0x7f7570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f7557fff640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f7557fff640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f7557fff640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.475+0000 7f7557fff640 10 client.4348.objecter ms_handle_connect 0x7f755c06e050 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.481+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7568050770 con 0x7f75700b17d0 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.481+0000 7f7557fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:43.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.481+0000 7f7557fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:43.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.860+0000 7f757590e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:43.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.860+0000 7f757590e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:43.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.860+0000 7f757590e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:22:43.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.860+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7550005330 con 0x7f75700b17d0 2024-03-29T11:22:43.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.861+0000 7f7557fff640 1 -- 172.21.15.19:0/891927209 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f75680526a0 con 0x7f75700b17d0 2024-03-29T11:22:43.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.861+0000 7f7557fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:43.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.861+0000 7f7557fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:43.839 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:43.839 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.865+0000 7f757590e640 10 librados: watch_flush enter 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.865+0000 7f757590e640 10 librados: watch_flush exit 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.865+0000 7f757590e640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.865+0000 7f757590e640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.866+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 >> v1:172.21.15.19:6816/3721232097 conn(0x7f755c06e050 legacy=0x7f755c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.866+0000 7f757590e640 10 monclient: shutdown 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.866+0000 7f757590e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.866+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 >> v1:172.21.15.62:6789/0 conn(0x7f75700b17d0 legacy=0x7f757006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.866+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 shutdown_connections 2024-03-29T11:22:43.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.866+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 >> 172.21.15.19:0/891927209 conn(0x7f7570066a00 msgr2=0x7f757013e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:43.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.867+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 shutdown_connections 2024-03-29T11:22:43.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.867+0000 7f757590e640 1 -- 172.21.15.19:0/891927209 wait complete. 2024-03-29T11:22:43.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:43.867+0000 7f757590e640 1 librados: shutdown 2024-03-29T11:22:43.858 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:50.665 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:50.816 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.838+0000 7efebc1ef640 10 monclient: get_monmap_and_config 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.838+0000 7efebc1ef640 10 monclient: build_initial_monmap 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.838+0000 7efebc1ef640 10 monclient: monmap: 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:50.840197+0000 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:50.840197+0000 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:50.817 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.818 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.839+0000 7efebc1ef640 1 Processor -- start 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.839+0000 7efebc1ef640 1 -- start start 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.839+0000 7efebc1ef640 10 monclient: init 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient(hunting): picked mon.noname-c con 0x7efeb41457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient(hunting): picked mon.noname-a con 0x7efeb413f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient(hunting): picked mon.noname-b con 0x7efeb40a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb40aaef0 con 0x7efeb413f650 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb4145bd0 con 0x7efeb41457f0 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb4145eb0 con 0x7efeb40a9d70 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efebc1ef640 10 monclient(hunting): authenticate will time out at 1091.104681s 2024-03-29T11:22:50.819 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efeb978a640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7efeb413f650 0x7efeb40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:45240/0 (socket says 172.21.15.19:45240) 2024-03-29T11:22:50.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efeba78c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7efeb40a9d70 0x7efeb414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:36006/0 (socket says 172.21.15.19:36006) 2024-03-29T11:22:50.820 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.840+0000 7efeba78c640 1 -- 172.21.15.19:0/1170663406 learned_addr learned my addr 172.21.15.19:0/1170663406 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:50.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb9f8b640 1 -- 172.21.15.19:0/1170663406 >> v1:172.21.15.62:6789/0 conn(0x7efeb41457f0 legacy=0x7efeb413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T11:22:50.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb9f8b640 1 -- 172.21.15.19:0/1170663406 >> v1:172.21.15.62:6789/0 conn(0x7efeb41457f0 legacy=0x7efeb413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:22:50.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb9f8b640 1 --1- 172.21.15.19:0/1170663406 >> v1:172.21.15.62:6789/0 conn(0x7efeb41457f0 0x7efeb413ef40 :-1 s=OPENED pgs=437678 cs=1 l=1).handle_message read tag failed 2024-03-29T11:22:50.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb9f8b640 1 --1- 172.21.15.19:0/1170663406 >> v1:172.21.15.62:6789/0 conn(0x7efeb41457f0 0x7efeb413ef40 :-1 s=OPENED pgs=437678 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:22:50.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.62:6789/0 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43740184 0 0) 0x7efeb4145eb0 con 0x7efeb40a9d70 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.841+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea0003520 con 0x7efeb40a9d70 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104256884 0 0) 0x7efeb40aaef0 con 0x7efeb413f650 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb4145eb0 con 0x7efeb413f650 2024-03-29T11:22:50.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472127908 0 0) 0x7efea0003520 con 0x7efeb40a9d70 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 >> v1:172.21.15.62:6789/0 conn(0x7efeb41457f0 legacy=0x7efeb413ef40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 >> v1:172.21.15.19:6789/0 conn(0x7efeb413f650 legacy=0x7efeb40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: _finish_hunting 0 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 monclient: found mon.noname-b 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb4146340 con 0x7efeb40a9d70 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: _finish_auth 0 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: _check_auth_tickets 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7efeac001a10 con 0x7efeb40a9d70 2024-03-29T11:22:50.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efeb8f89640 10 monclient: dump: 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:50.827 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efebc1ef640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.842+0000 7efebc1ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeac001c30 con 0x7efeb40a9d70 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efeb8f89640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efeb8f89640 1 -- 172.21.15.19:0/1170663406 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7efeac001fc0 con 0x7efeb40a9d70 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efeb8f89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efeb8f89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efeb8f89640 10 monclient: dump: 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.828 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efebc1ef640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.843+0000 7efebc1ef640 4 set_mon_vals no callback set 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 10 monclient: shutdown 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 1 -- 172.21.15.19:0/1170663406 >> v1:172.21.15.19:6790/0 conn(0x7efeb40a9d70 legacy=0x7efeb414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 1 -- 172.21.15.19:0/1170663406 shutdown_connections 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 1 -- 172.21.15.19:0/1170663406 >> 172.21.15.19:0/1170663406 conn(0x7efeb4066a00 msgr2=0x7efeb4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 1 -- 172.21.15.19:0/1170663406 shutdown_connections 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.844+0000 7efebc1ef640 1 -- 172.21.15.19:0/1170663406 wait complete. 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.845+0000 7efebc1ef640 10 monclient: build_initial_monmap 2024-03-29T11:22:50.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.845+0000 7efebc1ef640 10 monclient: monmap: 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 1 librados: starting msgr at 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 1 librados: starting objecter 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 1 Processor -- start 2024-03-29T11:22:50.830 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 1 -- start start 2024-03-29T11:22:50.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 1 librados: setting wanted keys 2024-03-29T11:22:50.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 1 librados: calling monclient init 2024-03-29T11:22:50.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.846+0000 7efebc1ef640 10 monclient: init 2024-03-29T11:22:50.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient(hunting): picked mon.noname-b con 0x7efeb40a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient(hunting): picked mon.noname-a con 0x7efeb413f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient(hunting): picked mon.noname-c con 0x7efeb41457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb4131a20 con 0x7efeb413f650 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efeb9f8b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7efeb40a9d70 0x7efeb41d65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:58156/0 (socket says 172.21.15.19:58156) 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efeb9f8b640 1 -- 172.21.15.19:0/3352982947 learned_addr learned my addr 172.21.15.19:0/3352982947 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb413d440 con 0x7efeb40a9d70 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb41317a0 con 0x7efeb41457f0 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efeaa7fc640 10 client.?.objecter ms_handle_connect 0x7efeb40a9d70 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:50.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efebc1ef640 10 monclient(hunting): authenticate will time out at 1091.111846s 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efeaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efeaa7fc640 10 client.?.objecter ms_handle_connect 0x7efeb41457f0 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.847+0000 7efeaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635953759 0 0) 0x7efeb41317a0 con 0x7efeb41457f0 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe98004eb0 con 0x7efeb41457f0 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 client.?.objecter ms_handle_connect 0x7efeb413f650 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949754647 0 0) 0x7efeb4131a20 con 0x7efeb413f650 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb41317a0 con 0x7efeb413f650 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418102133 0 0) 0x7efe98004eb0 con 0x7efeb41457f0 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb4131a20 con 0x7efeb41457f0 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7efeac001950 con 0x7efeb41457f0 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.836 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.848+0000 7efeaa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536693008 0 0) 0x7efeb413d440 con 0x7efeb40a9d70 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe98004eb0 con 0x7efeb40a9d70 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291809634 0 0) 0x7efeb41317a0 con 0x7efeb413f650 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efeb413d440 con 0x7efeb413f650 2024-03-29T11:22:50.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7efea40018b0 con 0x7efeb413f650 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient(hunting): dump: 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:50.838 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3819178412 0 0) 0x7efeb4131a20 con 0x7efeb41457f0 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 >> v1:172.21.15.62:6789/0 conn(0x7efeb40a9d70 legacy=0x7efeb41d65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 >> v1:172.21.15.19:6789/0 conn(0x7efeb413f650 legacy=0x7efeb4130980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 monclient: found mon.c 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb41411b0 con 0x7efeb41457f0 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efeaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:22:50.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 10 monclient: _renew_subs 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb4141490 con 0x7efeb41457f0 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 10 monclient: _renew_subs 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb4142d20 con 0x7efeb41457f0 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.849+0000 7efebc1ef640 1 librados: init done 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeac001ab0 con 0x7efeb41457f0 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7efeac004fb0 con 0x7efeb41457f0 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeaa7fc640 10 monclient: dump: 2024-03-29T11:22:50.840 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.850+0000 7efeabfff640 4 set_mon_vals no callback set 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.851+0000 7efebc1ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.851+0000 7efebc1ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.851+0000 7efebc1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:50.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.851+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe9c005080 con 0x7efeb41457f0 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.852+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7efeac01af80 con 0x7efeb41457f0 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7efeac084d20 con 0x7efeb41457f0 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeaa7fc640 10 client.4376.objecter ms_dispatch 0x7efeb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeaa7fc640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeaa7fc640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeb9f8b640 10 monclient: get_auth_request con 0x7efe9806e010 auth_method 0 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeaa7fc640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:22:50.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.853+0000 7efeaa7fc640 10 client.4376.objecter ms_handle_connect 0x7efe9806e010 2024-03-29T11:22:50.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.863+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efeac050430 con 0x7efeb41457f0 2024-03-29T11:22:50.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.863+0000 7efeaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:50.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:50.863+0000 7efeaa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:51.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.202+0000 7efebc1ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:51.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.202+0000 7efebc1ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:51.180 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.202+0000 7efebc1ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:51.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.202+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe9c005330 con 0x7efeb41457f0 2024-03-29T11:22:51.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.203+0000 7efeaa7fc640 1 -- 172.21.15.19:0/3352982947 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7efeac052530 con 0x7efeb41457f0 2024-03-29T11:22:51.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.203+0000 7efeaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:51.181 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.203+0000 7efeaa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:51.181 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:51.181 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 10 librados: watch_flush enter 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 10 librados: watch_flush exit 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 >> v1:172.21.15.19:6816/3721232097 conn(0x7efe9806e010 legacy=0x7efe980704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 10 monclient: shutdown 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 >> v1:172.21.15.19:6790/0 conn(0x7efeb41457f0 legacy=0x7efeb4131090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:51.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 shutdown_connections 2024-03-29T11:22:51.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.206+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 >> 172.21.15.19:0/3352982947 conn(0x7efeb4066a00 msgr2=0x7efeb40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:51.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.207+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 shutdown_connections 2024-03-29T11:22:51.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.207+0000 7efebc1ef640 1 -- 172.21.15.19:0/3352982947 wait complete. 2024-03-29T11:22:51.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:51.207+0000 7efebc1ef640 1 librados: shutdown 2024-03-29T11:22:51.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:58.005 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.172+0000 7fc5dea4e640 10 monclient: get_monmap_and_config 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.172+0000 7fc5dea4e640 10 monclient: build_initial_monmap 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.172+0000 7fc5dea4e640 10 monclient: monmap: 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:22:58.174027+0000 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:22:58.174027+0000 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:58.151 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:22:58.152 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:22:58.152 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:58.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.173+0000 7fc5dea4e640 1 Processor -- start 2024-03-29T11:22:58.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.174+0000 7fc5dea4e640 1 -- start start 2024-03-29T11:22:58.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.174+0000 7fc5dea4e640 10 monclient: init 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d80a9ac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d813ee90 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d80af900 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80afd10 con 0x7fc5d80af900 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80af6f0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80aac80 con 0x7fc5d813ee90 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5dea4e640 10 monclient(hunting): authenticate will time out at 1098.439474s 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5d7fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc5d80a9ac0 0x7fc5d80adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:37246/0 (socket says 172.21.15.19:37246) 2024-03-29T11:22:58.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.175+0000 7fc5d7fff640 1 -- 172.21.15.19:0/2390187629 learned_addr learned my addr 172.21.15.19:0/2390187629 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:58.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.177+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664231506 0 0) 0x7fc5d80af6f0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:58.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:58.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:58.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:22:58.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b8003520 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133700700 0 0) 0x7fc5d80aac80 con 0x7fc5d813ee90 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.178+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d80af6f0 con 0x7fc5d813ee90 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.179+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802025875 0 0) 0x7fc5b8003520 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.179+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 >> v1:172.21.15.19:6790/0 conn(0x7fc5d813ee90 legacy=0x7fc5d813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:58.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.179+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 >> v1:172.21.15.19:6789/0 conn(0x7fc5d80af900 legacy=0x7fc5d8149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:58.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:58.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 1 monclient: found mon.noname-c 2024-03-29T11:22:58.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:58.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d80aaf60 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5dea4e640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5dea4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc5c8001690 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5d77fe640 10 monclient: dump: 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:58.160 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:58.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.180+0000 7fc5dea4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5c80018b0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5dea4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5dea4e640 4 set_mon_vals no callback set 2024-03-29T11:22:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5d77fe640 1 -- 172.21.15.19:0/2390187629 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc5c8001c40 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.182+0000 7fc5d77fe640 10 monclient: dump: 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.183+0000 7fc5dea4e640 10 monclient: shutdown 2024-03-29T11:22:58.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.183+0000 7fc5dea4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.183+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2390187629 >> v1:172.21.15.62:6789/0 conn(0x7fc5d80a9ac0 legacy=0x7fc5d80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.183+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2390187629 shutdown_connections 2024-03-29T11:22:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.183+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2390187629 >> 172.21.15.19:0/2390187629 conn(0x7fc5d80b35e0 msgr2=0x7fc5d80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.184+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2390187629 shutdown_connections 2024-03-29T11:22:58.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.184+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2390187629 wait complete. 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 10 monclient: build_initial_monmap 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 10 monclient: monmap: 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 1 librados: starting msgr at 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 1 librados: starting objecter 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 1 Processor -- start 2024-03-29T11:22:58.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 1 -- start start 2024-03-29T11:22:58.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 1 librados: setting wanted keys 2024-03-29T11:22:58.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 1 librados: calling monclient init 2024-03-29T11:22:58.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.185+0000 7fc5dea4e640 10 monclient: init 2024-03-29T11:22:58.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:58.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:58.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient(hunting): picked mon.noname-c con 0x7fc5d80a9ac0 addr v1:172.21.15.19:6790/0 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient(hunting): picked mon.noname-b con 0x7fc5d80af900 addr v1:172.21.15.62:6789/0 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient(hunting): picked mon.noname-a con 0x7fc5d813ee90 addr v1:172.21.15.19:6789/0 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d81411e0 con 0x7fc5d813ee90 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8140ae0 con 0x7fc5d80af900 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d8140dc0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.186+0000 7fc5dea4e640 10 monclient(hunting): authenticate will time out at 1098.450859s 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.187+0000 7fc5cffff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc5d80af900 0x7fc5d8141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:39612/0 (socket says 172.21.15.19:39612) 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.187+0000 7fc5cffff640 1 -- 172.21.15.19:0/2015742048 learned_addr learned my addr 172.21.15.19:0/2015742048 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.187+0000 7fc5d57fa640 10 client.?.objecter ms_handle_connect 0x7fc5d80a9ac0 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.187+0000 7fc5d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643781921 0 0) 0x7fc5d8140dc0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:58.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc004eb0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1730090167 0 0) 0x7fc5bc004eb0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5d8140dc0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 client.?.objecter ms_handle_connect 0x7fc5d80af900 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc5c80015d0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.169 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.188+0000 7fc5d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710157649 0 0) 0x7fc5d8140ae0 con 0x7fc5d80af900 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5bc004eb0 con 0x7fc5d80af900 2024-03-29T11:22:58.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2970657821 0 0) 0x7fc5d8140dc0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 >> v1:172.21.15.62:6789/0 conn(0x7fc5d80af900 legacy=0x7fc5d8141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:58.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.189+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 >> v1:172.21.15.19:6789/0 conn(0x7fc5d813ee90 legacy=0x7fc5d8138560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:58.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.191+0000 7fc5d57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:58.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 1 monclient: found mon.c 2024-03-29T11:22:58.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:58.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:58.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d81ca260 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 10 monclient: _renew_subs 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d813f830 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 10 monclient: _renew_subs 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d8138c70 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5dea4e640 1 librados: init done 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5c8004b00 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc5c8004e50 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d6ffd640 4 set_mon_vals no callback set 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.192+0000 7fc5d57fa640 10 monclient: dump: 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:22:58.174 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:22:58.175 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.193+0000 7fc5dea4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.194+0000 7fc5dea4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.194+0000 7fc5dea4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.194+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b0005080 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.195+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fc5c8004fb0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5cffff640 10 monclient: get_auth_request con 0x7fc5bc06dd00 auth_method 0 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fc5c8084b10 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5d57fa640 10 client.4382.objecter ms_dispatch 0x7fc5d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5d57fa640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5d57fa640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5d57fa640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:22:58.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.196+0000 7fc5d57fa640 10 client.4382.objecter ms_handle_connect 0x7fc5bc06dd00 2024-03-29T11:22:58.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.205+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5c80503f0 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.205+0000 7fc5d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:58.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.205+0000 7fc5d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:58.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.547+0000 7fc5dea4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:58.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.547+0000 7fc5dea4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:58.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.547+0000 7fc5dea4e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:22:58.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.547+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b0005330 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.548+0000 7fc5d57fa640 1 -- 172.21.15.19:0/2015742048 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fc5c8052320 con 0x7fc5d80a9ac0 2024-03-29T11:22:58.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.548+0000 7fc5d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:58.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.548+0000 7fc5d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:58.529 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:22:58.530 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:58.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 10 librados: watch_flush enter 2024-03-29T11:22:58.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 10 librados: watch_flush exit 2024-03-29T11:22:58.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:22:58.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:22:58.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 >> v1:172.21.15.19:6816/3721232097 conn(0x7fc5bc06dd00 legacy=0x7fc5bc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:58.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 10 monclient: shutdown 2024-03-29T11:22:58.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:58.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.555+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 >> v1:172.21.15.19:6790/0 conn(0x7fc5d80a9ac0 legacy=0x7fc5d81416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:58.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.556+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 shutdown_connections 2024-03-29T11:22:58.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.556+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 >> 172.21.15.19:0/2015742048 conn(0x7fc5d80b35e0 msgr2=0x7fc5d81cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:58.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.556+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 shutdown_connections 2024-03-29T11:22:58.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.556+0000 7fc5dea4e640 1 -- 172.21.15.19:0/2015742048 wait complete. 2024-03-29T11:22:58.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:22:58.556+0000 7fc5dea4e640 1 librados: shutdown 2024-03-29T11:22:58.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:05.358 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:05.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.525+0000 7fd2a9b32640 10 monclient: get_monmap_and_config 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.525+0000 7fd2a9b32640 10 monclient: build_initial_monmap 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.525+0000 7fd2a9b32640 10 monclient: monmap: 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:05.527400+0000 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:05.527400+0000 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:05.505 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.526+0000 7fd2a9b32640 1 Processor -- start 2024-03-29T11:23:05.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.526+0000 7fd2a9b32640 1 -- start start 2024-03-29T11:23:05.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.526+0000 7fd2a9b32640 10 monclient: init 2024-03-29T11:23:05.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:05.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:05.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient(hunting): picked mon.noname-c con 0x7fd2a40b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient(hunting): picked mon.noname-b con 0x7fd2a414a700 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient(hunting): picked mon.noname-a con 0x7fd2a40ac5a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40aba70 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40aecb0 con 0x7fd2a40b0770 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40af150 con 0x7fd2a414a700 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.527+0000 7fd2a9b32640 10 monclient(hunting): authenticate will time out at 1105.791755s 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.528+0000 7fd2a37fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd2a40b0770 0x7fd2a4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:56504/0 (socket says 172.21.15.19:56504) 2024-03-29T11:23:05.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.528+0000 7fd2a37fe640 1 -- 172.21.15.19:0/2579739867 learned_addr learned my addr 172.21.15.19:0/2579739867 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.529+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377483562 0 0) 0x7fd2a40aecb0 con 0x7fd2a40b0770 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28c003520 con 0x7fd2a40b0770 2024-03-29T11:23:05.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738326140 0 0) 0x7fd28c003520 con 0x7fd2a40b0770 2024-03-29T11:23:05.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.530+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 >> v1:172.21.15.19:6790/0 conn(0x7fd2a414a700 legacy=0x7fd2a40abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:05.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.532+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 >> v1:172.21.15.19:6789/0 conn(0x7fd2a40ac5a0 legacy=0x7fd2a4151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.532+0000 7fd2a27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 1 monclient: found mon.noname-c 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40ac9d0 con 0x7fd2a40b0770 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a9b32640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:23:05.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a9b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:05.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd2940016b0 con 0x7fd2a40b0770 2024-03-29T11:23:05.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:05.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: dump: 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd294001940 con 0x7fd2a40b0770 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 1 -- 172.21.15.19:0/2579739867 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd294001cd0 con 0x7fd2a40b0770 2024-03-29T11:23:05.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a27fc640 10 monclient: dump: 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a9b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.533+0000 7fd2a9b32640 4 set_mon_vals no callback set 2024-03-29T11:23:05.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 10 monclient: shutdown 2024-03-29T11:23:05.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:05.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 1 -- 172.21.15.19:0/2579739867 >> v1:172.21.15.62:6789/0 conn(0x7fd2a40b0770 legacy=0x7fd2a4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 1 -- 172.21.15.19:0/2579739867 shutdown_connections 2024-03-29T11:23:05.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 1 -- 172.21.15.19:0/2579739867 >> 172.21.15.19:0/2579739867 conn(0x7fd2a406b660 msgr2=0x7fd2a406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:05.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 1 -- 172.21.15.19:0/2579739867 shutdown_connections 2024-03-29T11:23:05.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.535+0000 7fd2a9b32640 1 -- 172.21.15.19:0/2579739867 wait complete. 2024-03-29T11:23:05.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.536+0000 7fd2a9b32640 10 monclient: build_initial_monmap 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.536+0000 7fd2a9b32640 10 monclient: monmap: 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 1 librados: starting msgr at 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 1 librados: starting objecter 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 1 Processor -- start 2024-03-29T11:23:05.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 1 -- start start 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 1 librados: setting wanted keys 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 1 librados: calling monclient init 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.537+0000 7fd2a9b32640 10 monclient: init 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient(hunting): picked mon.noname-c con 0x7fd2a40ac5a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient(hunting): picked mon.noname-a con 0x7fd2a40b0770 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient(hunting): picked mon.noname-b con 0x7fd2a414a700 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a412a440 con 0x7fd2a40b0770 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a4128150 con 0x7fd2a414a700 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a4127d50 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:05.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.538+0000 7fd2a9b32640 10 monclient(hunting): authenticate will time out at 1105.802702s 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd2a3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd2a414a700 0x7fd2a41e5ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:33230/0 (socket says 172.21.15.19:33230) 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd2a3fff640 1 -- 172.21.15.19:0/679986310 learned_addr learned my addr 172.21.15.19:0/679986310 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 10 client.?.objecter ms_handle_connect 0x7fd2a414a700 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333382444 0 0) 0x7fd2a4128150 con 0x7fd2a414a700 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:23:05.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.540+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd290004eb0 con 0x7fd2a414a700 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.542+0000 7fd28bfff640 10 client.?.objecter ms_handle_connect 0x7fd2a40ac5a0 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.542+0000 7fd28bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 249867229 0 0) 0x7fd290004eb0 con 0x7fd2a414a700 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2a4128150 con 0x7fd2a414a700 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd29c001c60 con 0x7fd2a414a700 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:05.522 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 client.?.objecter ms_handle_connect 0x7fd2a40b0770 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:05.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540192662 0 0) 0x7fd2a4127d50 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.543+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd290005680 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897913878 0 0) 0x7fd2a412a440 con 0x7fd2a40b0770 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a4127d50 con 0x7fd2a40b0770 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2157521907 0 0) 0x7fd290005680 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2a412a440 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd294001570 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3807449986 0 0) 0x7fd2a4127d50 con 0x7fd2a40b0770 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd290005680 con 0x7fd2a40b0770 2024-03-29T11:23:05.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd2980016f0 con 0x7fd2a40b0770 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.544+0000 7fd28bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:05.526 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2456677055 0 0) 0x7fd2a412a440 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 >> v1:172.21.15.62:6789/0 conn(0x7fd2a414a700 legacy=0x7fd2a41e5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 >> v1:172.21.15.19:6789/0 conn(0x7fd2a40b0770 legacy=0x7fd2a4128cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 1 monclient: found mon.c 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a4147b60 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd28bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 10 monclient: _renew_subs 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a41df570 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 10 monclient: _renew_subs 2024-03-29T11:23:05.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a41ded30 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.545+0000 7fd2a9b32640 1 librados: init done 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.547+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd294001250 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.547+0000 7fd28bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.547+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd294004c30 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.547+0000 7fd28bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.548+0000 7fd28bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.548+0000 7fd28bfff640 10 monclient: dump: 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:05.529 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:05.530 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:05.530 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:05.530 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.548+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fd29401ac00 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd2a1ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fd294085be0 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd28bfff640 10 client.4385.objecter ms_dispatch 0x7fd2a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd28bfff640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd28bfff640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:05.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd28bfff640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T11:23:05.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd2a9b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:05.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.549+0000 7fd2a9b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:05.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.550+0000 7fd2a9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:05.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.550+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd284005080 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.550+0000 7fd2a2ffd640 10 monclient: get_auth_request con 0x7fd29006e6b0 auth_method 0 2024-03-29T11:23:05.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.550+0000 7fd28bfff640 10 client.4385.objecter ms_handle_connect 0x7fd29006e6b0 2024-03-29T11:23:05.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.558+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd294050340 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.558+0000 7fd28bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:05.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.558+0000 7fd28bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:05.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.891+0000 7fd2a9b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:05.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.891+0000 7fd2a9b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:05.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.891+0000 7fd2a9b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:05.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.891+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd284005330 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.893+0000 7fd28bfff640 1 -- 172.21.15.19:0/679986310 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fd294052440 con 0x7fd2a40ac5a0 2024-03-29T11:23:05.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.893+0000 7fd28bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:05.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.893+0000 7fd28bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:05.873 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:05.873 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:05.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.897+0000 7fd2a9b32640 10 librados: watch_flush enter 2024-03-29T11:23:05.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 10 librados: watch_flush exit 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 10 client.4385.objecter successfully canceled tick 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 >> v1:172.21.15.19:6816/3721232097 conn(0x7fd29006e6b0 legacy=0x7fd290070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 10 monclient: shutdown 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.898+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 >> v1:172.21.15.19:6790/0 conn(0x7fd2a40ac5a0 legacy=0x7fd2a41285e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.899+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 shutdown_connections 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.899+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 >> 172.21.15.19:0/679986310 conn(0x7fd2a406b660 msgr2=0x7fd2a40b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.900+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 shutdown_connections 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.900+0000 7fd2a9b32640 1 -- 172.21.15.19:0/679986310 wait complete. 2024-03-29T11:23:05.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:05.900+0000 7fd2a9b32640 1 librados: shutdown 2024-03-29T11:23:05.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:12.704 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:12.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.880+0000 7fae42156640 10 monclient: get_monmap_and_config 2024-03-29T11:23:12.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.880+0000 7fae42156640 10 monclient: build_initial_monmap 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.880+0000 7fae42156640 10 monclient: monmap: 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:12.881985+0000 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:12.881985+0000 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:12.860 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:12.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.881+0000 7fae42156640 1 Processor -- start 2024-03-29T11:23:12.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.881+0000 7fae42156640 1 -- start start 2024-03-29T11:23:12.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.881+0000 7fae42156640 10 monclient: init 2024-03-29T11:23:12.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:12.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:12.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient(hunting): picked mon.noname-c con 0x7fae3c0af0f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient(hunting): picked mon.noname-b con 0x7fae3c0af4d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient(hunting): picked mon.noname-a con 0x7fae3c0ab170 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c13f010 con 0x7fae3c0ab170 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0ab6c0 con 0x7fae3c0af0f0 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0ab9a0 con 0x7fae3c0af4d0 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.882+0000 7fae42156640 10 monclient(hunting): authenticate will time out at 1113.146469s 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.887+0000 7fae3bfff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fae3c0ab170 0x7fae3c14ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:51887/0 (socket says 172.21.15.19:51887) 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.887+0000 7fae3bfff640 1 -- 172.21.15.19:0/3371303790 learned_addr learned my addr 172.21.15.19:0/3371303790 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.888+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752519236 0 0) 0x7fae3c13f010 con 0x7fae3c0ab170 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.888+0000 7fae3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.888+0000 7fae3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.888+0000 7fae3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.888+0000 7fae3affd640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:23:12.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.888+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae28003520 con 0x7fae3c0ab170 2024-03-29T11:23:12.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781931666 0 0) 0x7fae3c0ab6c0 con 0x7fae3c0af0f0 2024-03-29T11:23:12.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c13f010 con 0x7fae3c0af0f0 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861836496 0 0) 0x7fae3c13f010 con 0x7fae3c0af0f0 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 >> v1:172.21.15.19:6790/0 conn(0x7fae3c0af4d0 legacy=0x7fae3c13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 >> v1:172.21.15.19:6789/0 conn(0x7fae3c0ab170 legacy=0x7fae3c14ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 1 monclient: found mon.noname-c 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.890+0000 7fae3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0abc80 con 0x7fae3c0af0f0 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: _finish_auth 0 2024-03-29T11:23:12.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae42156640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae42156640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae2c001670 con 0x7fae3c0af0f0 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: dump: 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:12.873 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae42156640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 1 -- 172.21.15.19:0/3371303790 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae2c001890 con 0x7fae3c0af0f0 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae42156640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.891+0000 7fae42156640 4 set_mon_vals no callback set 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.892+0000 7fae42156640 10 monclient: shutdown 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.892+0000 7fae42156640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:12.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.892+0000 7fae42156640 1 -- 172.21.15.19:0/3371303790 >> v1:172.21.15.62:6789/0 conn(0x7fae3c0af0f0 legacy=0x7fae3c13c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.893+0000 7fae42156640 1 -- 172.21.15.19:0/3371303790 shutdown_connections 2024-03-29T11:23:12.875 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.893+0000 7fae42156640 1 -- 172.21.15.19:0/3371303790 >> 172.21.15.19:0/3371303790 conn(0x7fae3c0b35e0 msgr2=0x7fae3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:12.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.895+0000 7fae42156640 1 -- 172.21.15.19:0/3371303790 shutdown_connections 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.895+0000 7fae42156640 1 -- 172.21.15.19:0/3371303790 wait complete. 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.896+0000 7fae42156640 10 monclient: build_initial_monmap 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.896+0000 7fae42156640 10 monclient: monmap: 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:12.882 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.896+0000 7fae42156640 1 librados: starting msgr at 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.896+0000 7fae42156640 1 librados: starting objecter 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.896+0000 7fae42156640 1 Processor -- start 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 1 -- start start 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 1 librados: setting wanted keys 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 1 librados: calling monclient init 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient: init 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient(hunting): picked mon.noname-b con 0x7fae3c0ab170 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient(hunting): picked mon.noname-c con 0x7fae3c0af0f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient(hunting): picked mon.noname-a con 0x7fae3c14f2c0 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:12.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c150b90 con 0x7fae3c14f2c0 2024-03-29T11:23:12.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c150de0 con 0x7fae3c0ab170 2024-03-29T11:23:12.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c1510c0 con 0x7fae3c0af0f0 2024-03-29T11:23:12.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:12.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.897+0000 7fae42156640 10 monclient(hunting): authenticate will time out at 1113.161922s 2024-03-29T11:23:12.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.901+0000 7fae3bfff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fae3c14f2c0 0x7fae3c14f6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:51967/0 (socket says 172.21.15.19:51967) 2024-03-29T11:23:12.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.901+0000 7fae3bfff640 1 -- 172.21.15.19:0/367325762 learned_addr learned my addr 172.21.15.19:0/367325762 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:12.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.904+0000 7fae38ff9640 10 client.?.objecter ms_handle_connect 0x7fae3c14f2c0 2024-03-29T11:23:12.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:12.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061641783 0 0) 0x7fae3c150b90 con 0x7fae3c14f2c0 2024-03-29T11:23:12.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae1c004eb0 con 0x7fae3c14f2c0 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.905+0000 7fae38ff9640 10 client.?.objecter ms_handle_connect 0x7fae3c0af0f0 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 client.?.objecter ms_handle_connect 0x7fae3c0ab170 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374186920 0 0) 0x7fae3c1510c0 con 0x7fae3c0af0f0 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c150b90 con 0x7fae3c0af0f0 2024-03-29T11:23:12.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982959645 0 0) 0x7fae3c150de0 con 0x7fae3c0ab170 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c1510c0 con 0x7fae3c0ab170 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 636883308 0 0) 0x7fae1c004eb0 con 0x7fae3c14f2c0 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae3c150de0 con 0x7fae3c14f2c0 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae34001cd0 con 0x7fae3c14f2c0 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:12.889 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4008523326 0 0) 0x7fae3c1510c0 con 0x7fae3c0ab170 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae1c004eb0 con 0x7fae3c0ab170 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae2c003790 con 0x7fae3c0ab170 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:12.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.906+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 437383137 0 0) 0x7fae1c004eb0 con 0x7fae3c0ab170 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 >> v1:172.21.15.19:6790/0 conn(0x7fae3c0af0f0 legacy=0x7fae3c1349d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 >> v1:172.21.15.19:6789/0 conn(0x7fae3c14f2c0 legacy=0x7fae3c14f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 1 monclient: found mon.b 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c14fe00 con 0x7fae3c0ab170 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:23:12.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 10 monclient: _renew_subs 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c1350e0 con 0x7fae3c0ab170 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 10 monclient: _renew_subs 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c0726d0 con 0x7fae3c0ab170 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.907+0000 7fae42156640 1 librados: init done 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.910+0000 7fae31ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.910+0000 7fae31ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.910+0000 7fae31ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.910+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c1515e0 con 0x7fae3c0ab170 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae2c001210 con 0x7fae3c0ab170 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae38ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:12.896 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae2c001a00 con 0x7fae3c0ab170 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae38ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae38ff9640 10 monclient: dump: 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:12.897 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.911+0000 7fae3a7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.912+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fae2c019a10 con 0x7fae3c0ab170 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fae2c0836d0 con 0x7fae3c0ab170 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae38ff9640 10 client.4369.objecter ms_dispatch 0x7fae3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae38ff9640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae38ff9640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae38ff9640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae32ffd640 10 monclient: get_auth_request con 0x7fae1c06dfb0 auth_method 0 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.914+0000 7fae38ff9640 10 client.4369.objecter ms_handle_connect 0x7fae1c06dfb0 2024-03-29T11:23:12.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.918+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae2c04ede0 con 0x7fae3c0ab170 2024-03-29T11:23:12.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.918+0000 7fae38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:12.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:12.918+0000 7fae38ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:13.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.250+0000 7fae31ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:13.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.250+0000 7fae31ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:13.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.250+0000 7fae31ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:13.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.250+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae3c1397c0 con 0x7fae3c0ab170 2024-03-29T11:23:13.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.252+0000 7fae38ff9640 1 -- 172.21.15.19:0/367325762 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fae2c001b60 con 0x7fae3c0ab170 2024-03-29T11:23:13.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.252+0000 7fae38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:13.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.252+0000 7fae38ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:13.232 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:13.232 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:13.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.255+0000 7fae31ffb640 10 librados: watch_flush enter 2024-03-29T11:23:13.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.255+0000 7fae31ffb640 10 librados: watch_flush exit 2024-03-29T11:23:13.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T11:23:13.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 10 client.4369.objecter successfully canceled tick 2024-03-29T11:23:13.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 >> v1:172.21.15.19:6816/3721232097 conn(0x7fae1c06dfb0 legacy=0x7fae1c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:13.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 10 monclient: shutdown 2024-03-29T11:23:13.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:13.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 >> v1:172.21.15.62:6789/0 conn(0x7fae3c0ab170 legacy=0x7fae3c138e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:13.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.256+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 shutdown_connections 2024-03-29T11:23:13.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.257+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 >> 172.21.15.19:0/367325762 conn(0x7fae3c0b35e0 msgr2=0x7fae3c0ae770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:13.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.258+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 shutdown_connections 2024-03-29T11:23:13.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.261+0000 7fae31ffb640 1 -- 172.21.15.19:0/367325762 wait complete. 2024-03-29T11:23:13.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:13.263+0000 7fae31ffb640 1 librados: shutdown 2024-03-29T11:23:13.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:20.065 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:20.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.246+0000 7f403e8b3640 10 monclient: get_monmap_and_config 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.246+0000 7f403e8b3640 10 monclient: build_initial_monmap 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.246+0000 7f403e8b3640 10 monclient: monmap: 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:20.247113+0000 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:20.247113+0000 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.247+0000 7f403e8b3640 1 Processor -- start 2024-03-29T11:23:20.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.247+0000 7f403e8b3640 1 -- start start 2024-03-29T11:23:20.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.247+0000 7f403e8b3640 10 monclient: init 2024-03-29T11:23:20.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.248+0000 7f403e8b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:20.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.248+0000 7f403e8b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:20.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.248+0000 7f403e8b3640 10 monclient(hunting): picked mon.noname-a con 0x7f40380af6c0 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 10 monclient(hunting): picked mon.noname-c con 0x7f403814fb00 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 10 monclient(hunting): picked mon.noname-b con 0x7f4038153090 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380a94e0 con 0x7f40380af6c0 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380afb60 con 0x7f403814fb00 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380aab80 con 0x7f4038153090 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.251+0000 7f403e8b3640 10 monclient(hunting): authenticate will time out at 1120.514901s 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4037fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f40380af6c0 0x7f40380ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:34449/0 (socket says 172.21.15.19:34449) 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4037fff640 1 -- 172.21.15.19:0/2208894283 learned_addr learned my addr 172.21.15.19:0/2208894283 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:20.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221529649 0 0) 0x7f40380a94e0 con 0x7f40380af6c0 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4036ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4036ffd640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.255+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020003520 con 0x7f40380af6c0 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.256+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2605222855 0 0) 0x7f4020003520 con 0x7f40380af6c0 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.256+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 >> v1:172.21.15.19:6790/0 conn(0x7f4038153090 legacy=0x7f403815a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:20.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.256+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 >> v1:172.21.15.62:6789/0 conn(0x7f403814fb00 legacy=0x7f4038152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:20.238 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 1 monclient: found mon.noname-a 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380aa680 con 0x7f40380af6c0 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f403e8b3640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f403e8b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4030001a10 con 0x7f40380af6c0 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:20.239 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f4036ffd640 10 monclient: dump: 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.257+0000 7f403e8b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.258+0000 7f4036ffd640 1 -- 172.21.15.19:0/2208894283 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4030001ca0 con 0x7f40380af6c0 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.258+0000 7f4036ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:20.240 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.258+0000 7f403e8b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.258+0000 7f403e8b3640 4 set_mon_vals no callback set 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.259+0000 7f403e8b3640 10 monclient: shutdown 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.259+0000 7f403e8b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.259+0000 7f403e8b3640 1 -- 172.21.15.19:0/2208894283 >> v1:172.21.15.19:6789/0 conn(0x7f40380af6c0 legacy=0x7f40380ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.259+0000 7f403e8b3640 1 -- 172.21.15.19:0/2208894283 shutdown_connections 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.259+0000 7f403e8b3640 1 -- 172.21.15.19:0/2208894283 >> 172.21.15.19:0/2208894283 conn(0x7f40380b35e0 msgr2=0x7f40380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:20.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.261+0000 7f403e8b3640 1 -- 172.21.15.19:0/2208894283 shutdown_connections 2024-03-29T11:23:20.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.261+0000 7f403e8b3640 1 -- 172.21.15.19:0/2208894283 wait complete. 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.262+0000 7f403e8b3640 10 monclient: build_initial_monmap 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.262+0000 7f403e8b3640 10 monclient: monmap: 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:20.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.262+0000 7f403e8b3640 1 librados: starting msgr at 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.262+0000 7f403e8b3640 1 librados: starting objecter 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 1 Processor -- start 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 1 -- start start 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 1 librados: setting wanted keys 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 1 librados: calling monclient init 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 10 monclient: init 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.263+0000 7f403e8b3640 10 monclient(hunting): picked mon.noname-c con 0x7f40380af6c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 10 monclient(hunting): picked mon.noname-b con 0x7f403814fb00 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 10 monclient(hunting): picked mon.noname-a con 0x7f4038153090 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403813d540 con 0x7f4038153090 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403813d790 con 0x7f403814fb00 2024-03-29T11:23:20.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4038071780 con 0x7f40380af6c0 2024-03-29T11:23:20.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:20.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.264+0000 7f403e8b3640 10 monclient(hunting): authenticate will time out at 1120.527031s 2024-03-29T11:23:20.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4037fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f40380af6c0 0x7f4038070eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:59179/0 (socket says 172.21.15.19:59179) 2024-03-29T11:23:20.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4037fff640 1 -- 172.21.15.19:0/1814470536 learned_addr learned my addr 172.21.15.19:0/1814470536 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:20.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 10 client.?.objecter ms_handle_connect 0x7f40380af6c0 2024-03-29T11:23:20.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:20.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679312867 0 0) 0x7f4038071780 con 0x7f40380af6c0 2024-03-29T11:23:20.248 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.268+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4024004eb0 con 0x7f40380af6c0 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2836126102 0 0) 0x7f4024004eb0 con 0x7f40380af6c0 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4038071780 con 0x7f40380af6c0 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4030009270 con 0x7f40380af6c0 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:20.249 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 10 client.?.objecter ms_handle_connect 0x7f4038153090 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3185745477 0 0) 0x7f4038071780 con 0x7f40380af6c0 2024-03-29T11:23:20.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.269+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 >> v1:172.21.15.62:6789/0 conn(0x7f403814fb00 legacy=0x7f40381e3f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 >> v1:172.21.15.19:6789/0 conn(0x7f4038153090 legacy=0x7f403808a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 1 monclient: found mon.c 2024-03-29T11:23:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:20.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4038151f10 con 0x7f40380af6c0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f4034ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 10 monclient: _renew_subs 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40381521c0 con 0x7f40380af6c0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 10 monclient: _renew_subs 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f403808ad20 con 0x7f40380af6c0 2024-03-29T11:23:20.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.271+0000 7f403e8b3640 1 librados: init done 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4030009680 con 0x7f40380af6c0 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4034ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4030009ad0 con 0x7f40380af6c0 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4034ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4034ff9640 10 monclient: dump: 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:20.255 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f4035ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.273+0000 7f403e8b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.274+0000 7f403e8b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.274+0000 7f403e8b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.274+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4018005080 con 0x7f40380af6c0 2024-03-29T11:23:20.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.276+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f4030001e90 con 0x7f40380af6c0 2024-03-29T11:23:20.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.277+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f4030089570 con 0x7f40380af6c0 2024-03-29T11:23:20.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.277+0000 7f4034ff9640 10 client.4394.objecter ms_dispatch 0x7f4038000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:20.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.277+0000 7f4034ff9640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:20.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.277+0000 7f4034ff9640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:20.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.277+0000 7f4034ff9640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T11:23:20.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.280+0000 7f40377fe640 10 monclient: get_auth_request con 0x7f402406db10 auth_method 0 2024-03-29T11:23:20.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.280+0000 7f4034ff9640 10 client.4394.objecter ms_handle_connect 0x7f402406db10 2024-03-29T11:23:20.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.287+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4030054e50 con 0x7f40380af6c0 2024-03-29T11:23:20.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.287+0000 7f4034ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:20.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.287+0000 7f4034ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:20.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.613+0000 7f403e8b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:20.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.613+0000 7f403e8b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:20.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.613+0000 7f403e8b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:20.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.613+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4018005330 con 0x7f40380af6c0 2024-03-29T11:23:20.594 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.616+0000 7f4034ff9640 1 -- 172.21.15.19:0/1814470536 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f4030056d80 con 0x7f40380af6c0 2024-03-29T11:23:20.594 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.616+0000 7f4034ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:20.594 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.616+0000 7f4034ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:20.595 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:20.595 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:20.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 10 librados: watch_flush enter 2024-03-29T11:23:20.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 10 librados: watch_flush exit 2024-03-29T11:23:20.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T11:23:20.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 10 client.4394.objecter successfully canceled tick 2024-03-29T11:23:20.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 >> v1:172.21.15.19:6816/3721232097 conn(0x7f402406db10 legacy=0x7f402406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:20.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 10 monclient: shutdown 2024-03-29T11:23:20.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:20.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 >> v1:172.21.15.19:6790/0 conn(0x7f40380af6c0 legacy=0x7f4038070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:20.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 shutdown_connections 2024-03-29T11:23:20.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.620+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 >> 172.21.15.19:0/1814470536 conn(0x7f40380b35e0 msgr2=0x7f40380ad240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:20.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.622+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 shutdown_connections 2024-03-29T11:23:20.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.622+0000 7f403e8b3640 1 -- 172.21.15.19:0/1814470536 wait complete. 2024-03-29T11:23:20.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:20.624+0000 7f403e8b3640 1 librados: shutdown 2024-03-29T11:23:20.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:27.426 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:27.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.605+0000 7fe4a3fff640 10 monclient: get_monmap_and_config 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.605+0000 7fe4a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.605+0000 7fe4a3fff640 10 monclient: monmap: 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:27.606692+0000 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:27.606692+0000 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:27.587 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:27.588 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:27.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.609+0000 7fe4a3fff640 1 Processor -- start 2024-03-29T11:23:27.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.610+0000 7fe4a3fff640 1 -- start start 2024-03-29T11:23:27.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.610+0000 7fe4a3fff640 10 monclient: init 2024-03-29T11:23:27.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:27.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:27.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a40a9750 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:27.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a40adca0 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:27.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a4146a80 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:27.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40abdc0 con 0x7fe4a4146a80 2024-03-29T11:23:27.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40b0070 con 0x7fe4a40adca0 2024-03-29T11:23:27.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40af880 con 0x7fe4a40a9750 2024-03-29T11:23:27.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:27.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.611+0000 7fe4a3fff640 10 monclient(hunting): authenticate will time out at 1127.874360s 2024-03-29T11:23:27.594 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.614+0000 7fe4a2ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe4a40a9750 0x7fe4a4143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:49249/0 (socket says 172.21.15.19:49249) 2024-03-29T11:23:27.594 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.614+0000 7fe4a2ffd640 1 -- 172.21.15.19:0/669476759 learned_addr learned my addr 172.21.15.19:0/669476759 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:27.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.618+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857197845 0 0) 0x7fe4a40af880 con 0x7fe4a40a9750 2024-03-29T11:23:27.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.618+0000 7fe4a1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:27.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.618+0000 7fe4a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.618+0000 7fe4a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.618+0000 7fe4a1ffb640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.618+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe48c003520 con 0x7fe4a40a9750 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3799480670 0 0) 0x7fe48c003520 con 0x7fe4a40a9750 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 >> v1:172.21.15.62:6789/0 conn(0x7fe4a40adca0 legacy=0x7fe4a4146370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 >> v1:172.21.15.19:6789/0 conn(0x7fe4a4146a80 legacy=0x7fe4a4151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:27.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40ac030 con 0x7fe4a40a9750 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe49c001a10 con 0x7fe4a40a9750 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a1ffb640 10 monclient: dump: 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.599 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a3fff640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.619+0000 7fe4a3fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:27.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe49c001be0 con 0x7fe4a40a9750 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a1ffb640 1 -- 172.21.15.19:0/669476759 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe49c002030 con 0x7fe4a40a9750 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a1ffb640 10 monclient: dump: 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:27.601 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a3fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.620+0000 7fe4a3fff640 4 set_mon_vals no callback set 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.621+0000 7fe4a3fff640 10 monclient: shutdown 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.621+0000 7fe4a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:27.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.621+0000 7fe4a3fff640 1 -- 172.21.15.19:0/669476759 >> v1:172.21.15.19:6790/0 conn(0x7fe4a40a9750 legacy=0x7fe4a4143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.623+0000 7fe4a3fff640 1 -- 172.21.15.19:0/669476759 shutdown_connections 2024-03-29T11:23:27.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.623+0000 7fe4a3fff640 1 -- 172.21.15.19:0/669476759 >> 172.21.15.19:0/669476759 conn(0x7fe4a40673e0 msgr2=0x7fe4a40b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:27.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.623+0000 7fe4a3fff640 1 -- 172.21.15.19:0/669476759 shutdown_connections 2024-03-29T11:23:27.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.623+0000 7fe4a3fff640 1 -- 172.21.15.19:0/669476759 wait complete. 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.624+0000 7fe4a3fff640 10 monclient: build_initial_monmap 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.624+0000 7fe4a3fff640 10 monclient: monmap: 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:27.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.625+0000 7fe4a3fff640 1 librados: starting msgr at 2024-03-29T11:23:27.605 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.625+0000 7fe4a3fff640 1 librados: starting objecter 2024-03-29T11:23:27.605 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.625+0000 7fe4a3fff640 1 Processor -- start 2024-03-29T11:23:27.605 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.625+0000 7fe4a3fff640 1 -- start start 2024-03-29T11:23:27.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.625+0000 7fe4a3fff640 1 librados: setting wanted keys 2024-03-29T11:23:27.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.626+0000 7fe4a3fff640 1 librados: calling monclient init 2024-03-29T11:23:27.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.626+0000 7fe4a3fff640 10 monclient: init 2024-03-29T11:23:27.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.627+0000 7fe4a3fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:27.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.627+0000 7fe4a3fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.627+0000 7fe4a3fff640 10 monclient(hunting): picked mon.noname-b con 0x7fe4a40a9750 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 10 monclient(hunting): picked mon.noname-c con 0x7fe4a40adca0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 10 monclient(hunting): picked mon.noname-a con 0x7fe4a4146a80 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a41dc090 con 0x7fe4a4146a80 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a41db180 con 0x7fe4a40a9750 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a41db460 con 0x7fe4a40adca0 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:27.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.629+0000 7fe4a3fff640 10 monclient(hunting): authenticate will time out at 1127.892203s 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.633+0000 7fe4a2ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe4a40a9750 0x7fe4a41db980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:34315/0 (socket says 172.21.15.19:34315) 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.633+0000 7fe4a2ffd640 1 -- 172.21.15.19:0/1401554512 learned_addr learned my addr 172.21.15.19:0/1401554512 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 client.?.objecter ms_handle_connect 0x7fe4a40a9750 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 client.?.objecter ms_handle_connect 0x7fe4a4146a80 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528507375 0 0) 0x7fe4a41dc090 con 0x7fe4a4146a80 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:27.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.634+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe490004eb0 con 0x7fe4a4146a80 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 356118902 0 0) 0x7fe490004eb0 con 0x7fe4a4146a80 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4a41dc090 con 0x7fe4a4146a80 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe494001750 con 0x7fe4a4146a80 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient(hunting): dump: 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:27.619 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 669834600 0 0) 0x7fe4a41dc090 con 0x7fe4a4146a80 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 >> v1:172.21.15.19:6790/0 conn(0x7fe4a40adca0 legacy=0x7fe4a413e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 >> v1:172.21.15.62:6789/0 conn(0x7fe4a40a9750 legacy=0x7fe4a41db980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 monclient: found mon.a 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a4072520 con 0x7fe4a4146a80 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:27.620 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe494001ce0 con 0x7fe4a4146a80 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe4a17fa640 4 set_mon_vals no callback set 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe494003790 con 0x7fe4a4146a80 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.635+0000 7fe48b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe48b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe48b7fe640 10 monclient: dump: 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:27.621 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 10 monclient: _renew_subs 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a4072770 con 0x7fe4a4146a80 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 10 monclient: _renew_subs 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a413eaf0 con 0x7fe4a4146a80 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.636+0000 7fe4a3fff640 1 librados: init done 2024-03-29T11:23:27.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.638+0000 7fe4a3fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:27.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.639+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fe494019760 con 0x7fe4a4146a80 2024-03-29T11:23:27.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.639+0000 7fe4a3fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:27.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.639+0000 7fe4a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:27.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.639+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a41db5f0 con 0x7fe4a4146a80 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.639+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fe494083610 con 0x7fe4a4146a80 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.640+0000 7fe4a2ffd640 10 monclient: get_auth_request con 0x7fe49006dbe0 auth_method 0 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.640+0000 7fe48b7fe640 10 client.4377.objecter ms_dispatch 0x7fe4a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.640+0000 7fe48b7fe640 3 client.4377.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.640+0000 7fe48b7fe640 3 client.4377.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.640+0000 7fe48b7fe640 20 client.4377.objecter dump_active .. 0 homeless 2024-03-29T11:23:27.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.640+0000 7fe48b7fe640 10 client.4377.objecter ms_handle_connect 0x7fe49006dbe0 2024-03-29T11:23:27.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.650+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe49404eef0 con 0x7fe4a4146a80 2024-03-29T11:23:27.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.651+0000 7fe48b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:27.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.651+0000 7fe48b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:27.965 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.986+0000 7fe4a3fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:27.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.986+0000 7fe4a3fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:27.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.986+0000 7fe4a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:27.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.986+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4a404f730 con 0x7fe4a4146a80 2024-03-29T11:23:27.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.988+0000 7fe48b7fe640 1 -- 172.21.15.19:0/1401554512 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fe494050e20 con 0x7fe4a4146a80 2024-03-29T11:23:27.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.988+0000 7fe48b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:27.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.988+0000 7fe48b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:27.967 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:27.967 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:27.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.992+0000 7fe4a3fff640 10 librados: watch_flush enter 2024-03-29T11:23:27.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.992+0000 7fe4a3fff640 10 librados: watch_flush exit 2024-03-29T11:23:27.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.992+0000 7fe4a3fff640 20 client.4377.objecter shutdown clearing up homeless session... 2024-03-29T11:23:27.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.992+0000 7fe4a3fff640 10 client.4377.objecter successfully canceled tick 2024-03-29T11:23:27.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.993+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 >> v1:172.21.15.19:6816/3721232097 conn(0x7fe49006dbe0 legacy=0x7fe4900700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.993+0000 7fe4a3fff640 10 monclient: shutdown 2024-03-29T11:23:27.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.993+0000 7fe4a3fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:27.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.993+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 >> v1:172.21.15.19:6789/0 conn(0x7fe4a4146a80 legacy=0x7fe4a41e6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:27.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.993+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 shutdown_connections 2024-03-29T11:23:27.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.993+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 >> 172.21.15.19:0/1401554512 conn(0x7fe4a40673e0 msgr2=0x7fe4a40aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:27.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.995+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 shutdown_connections 2024-03-29T11:23:27.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.995+0000 7fe4a3fff640 1 -- 172.21.15.19:0/1401554512 wait complete. 2024-03-29T11:23:27.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:27.996+0000 7fe4a3fff640 1 librados: shutdown 2024-03-29T11:23:27.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:34.805 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.973+0000 7f00637d6640 10 monclient: get_monmap_and_config 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.973+0000 7f00637d6640 10 monclient: build_initial_monmap 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.973+0000 7f00637d6640 10 monclient: monmap: 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:34.974586+0000 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:34.974586+0000 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:34.954 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:34.955 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:34.955 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:34.955 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:34.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.974+0000 7f00637d6640 1 Processor -- start 2024-03-29T11:23:34.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.975+0000 7f00637d6640 1 -- start start 2024-03-29T11:23:34.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.975+0000 7f00637d6640 10 monclient: init 2024-03-29T11:23:34.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.976+0000 7f00637d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:34.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.976+0000 7f00637d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:34.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.976+0000 7f00637d6640 10 monclient(hunting): picked mon.noname-b con 0x7f005c0a9ac0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:34.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.976+0000 7f00637d6640 10 monclient(hunting): picked mon.noname-a con 0x7f005c13ee90 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:34.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.979+0000 7f00637d6640 10 monclient(hunting): picked mon.noname-c con 0x7f005c0af900 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:34.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f00637d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0afd10 con 0x7f005c13ee90 2024-03-29T11:23:34.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f00637d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0af6f0 con 0x7f005c0af900 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f00637d6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0aac80 con 0x7f005c0a9ac0 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0060d71640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f005c13ee90 0x7f005c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:42943/0 (socket says 172.21.15.19:42943) 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0060d71640 1 -- 172.21.15.19:0/158102367 learned_addr learned my addr 172.21.15.19:0/158102367 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f00637d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f00637d6640 10 monclient(hunting): authenticate will time out at 1135.246426s 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.980+0000 7f0061572640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f005c0a9ac0 0x7f005c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46025/0 (socket says 172.21.15.19:46025) 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539019215 0 0) 0x7f005c0afd10 con 0x7f005c13ee90 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0053fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0053fff640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:23:34.965 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.983+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0048003520 con 0x7f005c13ee90 2024-03-29T11:23:34.965 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3219368569 0 0) 0x7f005c0aac80 con 0x7f005c0a9ac0 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c0afd10 con 0x7f005c0a9ac0 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525670109 0 0) 0x7f005c0af6f0 con 0x7f005c0af900 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:23:34.966 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f005c0aac80 con 0x7f005c0af900 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 912890836 0 0) 0x7f005c0afd10 con 0x7f005c0a9ac0 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 >> v1:172.21.15.62:6789/0 conn(0x7f005c0af900 legacy=0x7f005c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 >> v1:172.21.15.19:6789/0 conn(0x7f005c13ee90 legacy=0x7f005c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 monclient: found mon.noname-b 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0aaf60 con 0x7f005c0a9ac0 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:34.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0058001670 con 0x7f005c0a9ac0 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.984+0000 7f0053fff640 10 monclient: dump: 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:34.968 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00580017d0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f0053fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f0053fff640 1 -- 172.21.15.19:0/158102367 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0058001be0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f0053fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f0053fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f0053fff640 10 monclient: dump: 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.969 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f00637d6640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f00637d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.985+0000 7f00637d6640 4 set_mon_vals no callback set 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.986+0000 7f00637d6640 10 monclient: shutdown 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.986+0000 7f00637d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:34.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.986+0000 7f00637d6640 1 -- 172.21.15.19:0/158102367 >> v1:172.21.15.19:6790/0 conn(0x7f005c0a9ac0 legacy=0x7f005c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.987+0000 7f00637d6640 1 -- 172.21.15.19:0/158102367 shutdown_connections 2024-03-29T11:23:34.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.987+0000 7f00637d6640 1 -- 172.21.15.19:0/158102367 >> 172.21.15.19:0/158102367 conn(0x7f005c0b35e0 msgr2=0x7f005c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:34.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.987+0000 7f00637d6640 1 -- 172.21.15.19:0/158102367 shutdown_connections 2024-03-29T11:23:34.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.987+0000 7f00637d6640 1 -- 172.21.15.19:0/158102367 wait complete. 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 10 monclient: build_initial_monmap 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 10 monclient: monmap: 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:34.974 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 1 librados: starting msgr at 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 1 librados: starting objecter 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 1 Processor -- start 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 1 -- start start 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 1 librados: setting wanted keys 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 1 librados: calling monclient init 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.989+0000 7f00637d6640 10 monclient: init 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.990+0000 7f00637d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.990+0000 7f00637d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.990+0000 7f00637d6640 10 monclient(hunting): picked mon.noname-b con 0x7f005c0a9ac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:34.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 10 monclient(hunting): picked mon.noname-c con 0x7f005c0af900 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:34.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 10 monclient(hunting): picked mon.noname-a con 0x7f005c13ee90 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:34.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c1d6800 con 0x7f005c13ee90 2024-03-29T11:23:34.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c1c66a0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c1d73f0 con 0x7f005c0af900 2024-03-29T11:23:34.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:34.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.992+0000 7f00637d6640 10 monclient(hunting): authenticate will time out at 1135.255727s 2024-03-29T11:23:34.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.994+0000 7f0061572640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f005c0a9ac0 0x7f005c1d6ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:36163/0 (socket says 172.21.15.19:36163) 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.994+0000 7f0061572640 1 -- 172.21.15.19:0/2579615488 learned_addr learned my addr 172.21.15.19:0/2579615488 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.997+0000 7f0051ffb640 10 client.?.objecter ms_handle_connect 0x7f005c0a9ac0 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.997+0000 7f0051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163939710 0 0) 0x7f005c1c66a0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0040004eb0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.981 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 client.?.objecter ms_handle_connect 0x7f005c13ee90 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 69536233 0 0) 0x7f0040004eb0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f005c1c66a0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0058001c20 con 0x7f005c0a9ac0 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:34.982 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028488964 0 0) 0x7f005c1d6800 con 0x7f005c13ee90 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:34.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.998+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0040004eb0 con 0x7f005c13ee90 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2701826035 0 0) 0x7f005c1c66a0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 >> v1:172.21.15.19:6790/0 conn(0x7f005c0af900 legacy=0x7f005c1377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 >> v1:172.21.15.19:6789/0 conn(0x7f005c13ee90 legacy=0x7f005c1dea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 1 monclient: found mon.b 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c1d7580 con 0x7f005c0a9ac0 2024-03-29T11:23:34.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f0051ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 10 monclient: _renew_subs 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c1c5060 con 0x7f005c0a9ac0 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 10 monclient: _renew_subs 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c0701a0 con 0x7f005c0a9ac0 2024-03-29T11:23:34.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:34.999+0000 7f00637d6640 1 librados: init done 2024-03-29T11:23:34.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.000+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0058004b10 con 0x7f005c0a9ac0 2024-03-29T11:23:34.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.000+0000 7f0051ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:34.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.000+0000 7f00537fe640 4 set_mon_vals no callback set 2024-03-29T11:23:34.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.001+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0058004e60 con 0x7f005c0a9ac0 2024-03-29T11:23:34.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f0051ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:34.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f0051ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f0051ffb640 10 monclient: dump: 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f0058005080 con 0x7f005c0a9ac0 2024-03-29T11:23:34.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f00637d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f00637d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f00637d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.002+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003c005080 con 0x7f005c0a9ac0 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.003+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f0058084d60 con 0x7f005c0a9ac0 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.004+0000 7f0060d71640 10 monclient: get_auth_request con 0x7f004006dd00 auth_method 0 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.004+0000 7f0051ffb640 10 client.4375.objecter ms_dispatch 0x7f005c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.004+0000 7f0051ffb640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.004+0000 7f0051ffb640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.004+0000 7f0051ffb640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:23:34.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.004+0000 7f0051ffb640 10 client.4375.objecter ms_handle_connect 0x7f004006dd00 2024-03-29T11:23:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.010+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0058050640 con 0x7f005c0a9ac0 2024-03-29T11:23:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.010+0000 7f0051ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:34.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.010+0000 7f0051ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:35.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.336+0000 7f00637d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:35.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.336+0000 7f00637d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:35.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.336+0000 7f00637d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:35.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.336+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f003c005330 con 0x7f005c0a9ac0 2024-03-29T11:23:35.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.337+0000 7f0051ffb640 1 -- 172.21.15.19:0/2579615488 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f0058052570 con 0x7f005c0a9ac0 2024-03-29T11:23:35.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.337+0000 7f0051ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:35.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.337+0000 7f0051ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:35.317 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:35.317 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:35.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 10 librados: watch_flush enter 2024-03-29T11:23:35.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 10 librados: watch_flush exit 2024-03-29T11:23:35.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:23:35.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:23:35.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 >> v1:172.21.15.19:6816/3721232097 conn(0x7f004006dd00 legacy=0x7f00400701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:35.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 10 monclient: shutdown 2024-03-29T11:23:35.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:35.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 >> v1:172.21.15.62:6789/0 conn(0x7f005c0a9ac0 legacy=0x7f005c1d6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:35.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 shutdown_connections 2024-03-29T11:23:35.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.341+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 >> 172.21.15.19:0/2579615488 conn(0x7f005c0b35e0 msgr2=0x7f005c13af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:35.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.342+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 shutdown_connections 2024-03-29T11:23:35.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.342+0000 7f00637d6640 1 -- 172.21.15.19:0/2579615488 wait complete. 2024-03-29T11:23:35.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:35.342+0000 7f00637d6640 1 librados: shutdown 2024-03-29T11:23:35.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:42.145 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:42.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.315+0000 7f5f63889640 10 monclient: get_monmap_and_config 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.315+0000 7f5f63889640 10 monclient: build_initial_monmap 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.316+0000 7f5f63889640 10 monclient: monmap: 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:42.316799+0000 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:42.316799+0000 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:42.296 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.319+0000 7f5f63889640 1 Processor -- start 2024-03-29T11:23:42.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.319+0000 7f5f63889640 1 -- start start 2024-03-29T11:23:42.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.319+0000 7f5f63889640 10 monclient: init 2024-03-29T11:23:42.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:42.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c1457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c13f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c0a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c0aaef0 con 0x7f5f5c13f650 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c145bd0 con 0x7f5f5c1457f0 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c145eb0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.320+0000 7f5f63889640 10 monclient(hunting): authenticate will time out at 1142.583587s 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.321+0000 7f5f61e26640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f5f5c0a9d70 0x7f5f5c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:45552/0 (socket says 172.21.15.19:45552) 2024-03-29T11:23:42.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.321+0000 7f5f61e26640 1 -- 172.21.15.19:0/69868708 learned_addr learned my addr 172.21.15.19:0/69868708 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:42.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142341324 0 0) 0x7f5f5c145eb0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:42.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:42.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:42.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:23:42.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f48003520 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058001672 0 0) 0x7f5f48003520 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.322+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 >> v1:172.21.15.62:6789/0 conn(0x7f5f5c1457f0 legacy=0x7f5f5c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:42.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 >> v1:172.21.15.19:6789/0 conn(0x7f5f5c13f650 legacy=0x7f5f5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:42.304 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 1 monclient: found mon.noname-b 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c146340 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5f4c001690 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f53fff640 10 monclient: dump: 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:42.305 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f63889640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:23:42.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.324+0000 7f5f63889640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:42.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.327+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f4c001860 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.327+0000 7f5f53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:42.307 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.327+0000 7f5f53fff640 1 -- 172.21.15.19:0/69868708 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5f4c001cb0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.327+0000 7f5f63889640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.327+0000 7f5f63889640 4 set_mon_vals no callback set 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f53fff640 10 monclient: dump: 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:42.308 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f63889640 10 monclient: shutdown 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f63889640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f63889640 1 -- 172.21.15.19:0/69868708 >> v1:172.21.15.19:6790/0 conn(0x7f5f5c0a9d70 legacy=0x7f5f5c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f63889640 1 -- 172.21.15.19:0/69868708 shutdown_connections 2024-03-29T11:23:42.309 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.328+0000 7f5f63889640 1 -- 172.21.15.19:0/69868708 >> 172.21.15.19:0/69868708 conn(0x7f5f5c066a00 msgr2=0x7f5f5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:42.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.333+0000 7f5f63889640 1 -- 172.21.15.19:0/69868708 shutdown_connections 2024-03-29T11:23:42.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.333+0000 7f5f63889640 1 -- 172.21.15.19:0/69868708 wait complete. 2024-03-29T11:23:42.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.334+0000 7f5f63889640 10 monclient: build_initial_monmap 2024-03-29T11:23:42.315 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.334+0000 7f5f63889640 10 monclient: monmap: 2024-03-29T11:23:42.315 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:42.315 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:42.315 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 1 librados: starting msgr at 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 1 librados: starting objecter 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 1 Processor -- start 2024-03-29T11:23:42.316 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 1 -- start start 2024-03-29T11:23:42.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 1 librados: setting wanted keys 2024-03-29T11:23:42.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 1 librados: calling monclient init 2024-03-29T11:23:42.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.335+0000 7f5f63889640 10 monclient: init 2024-03-29T11:23:42.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.336+0000 7f5f63889640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.336+0000 7f5f63889640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f61625640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f5f5c0a9d70 0x7f5f5c136290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54318/0 (socket says 172.21.15.19:54318) 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f61625640 1 -- 172.21.15.19:0/4034866954 learned_addr learned my addr 172.21.15.19:0/4034866954 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 10 monclient(hunting): picked mon.noname-c con 0x7f5f5c0a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 10 monclient(hunting): picked mon.noname-b con 0x7f5f5c13f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 10 monclient(hunting): picked mon.noname-a con 0x7f5f5c1457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c143ec0 con 0x7f5f5c1457f0 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c1434e0 con 0x7f5f5c13f650 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5c1437c0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:42.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f63889640 10 monclient(hunting): authenticate will time out at 1142.600471s 2024-03-29T11:23:42.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f51ffb640 10 client.?.objecter ms_handle_connect 0x7f5f5c0a9d70 2024-03-29T11:23:42.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.337+0000 7f5f51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 client.?.objecter ms_handle_connect 0x7f5f5c13f650 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 client.?.objecter ms_handle_connect 0x7f5f5c1457f0 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509876994 0 0) 0x7f5f5c1437c0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f40004eb0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.325 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132140253 0 0) 0x7f5f5c1434e0 con 0x7f5f5c13f650 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5c1437c0 con 0x7f5f5c13f650 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305855997 0 0) 0x7f5f5c143ec0 con 0x7f5f5c1457f0 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:23:42.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.341+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f5c1434e0 con 0x7f5f5c1457f0 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2420689191 0 0) 0x7f5f40004eb0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f5c143ec0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5f580017d0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:42.327 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 470013900 0 0) 0x7f5f5c1437c0 con 0x7f5f5c13f650 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f40004eb0 con 0x7f5f5c13f650 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5f54001ba0 con 0x7f5f5c13f650 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.328 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2918506077 0 0) 0x7f5f5c143ec0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 >> v1:172.21.15.62:6789/0 conn(0x7f5f5c13f650 legacy=0x7f5f5c130c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 >> v1:172.21.15.19:6789/0 conn(0x7f5f5c1457f0 legacy=0x7f5f5c144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 monclient: found mon.c 2024-03-29T11:23:42.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5c1369a0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f58001930 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f63889640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f63889640 10 monclient: _renew_subs 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f63889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.342+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5c136bf0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5f58004a00 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f63889640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f63889640 10 monclient: _renew_subs 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f63889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5c131550 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f63889640 1 librados: init done 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.343+0000 7f5f51ffb640 10 monclient: dump: 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:42.331 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.344+0000 7f5f537fe640 4 set_mon_vals no callback set 2024-03-29T11:23:42.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.345+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f5f5801a9d0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f60e24640 10 monclient: get_auth_request con 0x7f5f4006e000 auth_method 0 2024-03-29T11:23:42.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f63889640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:42.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f63889640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:42.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f63889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:42.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3c005080 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f5f580849b0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f51ffb640 10 client.4406.objecter ms_dispatch 0x7f5f5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:42.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f51ffb640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:42.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f51ffb640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:42.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.346+0000 7f5f51ffb640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:23:42.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.347+0000 7f5f51ffb640 10 client.4406.objecter ms_handle_connect 0x7f5f4006e000 2024-03-29T11:23:42.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.355+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f580500c0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.355+0000 7f5f51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:42.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.355+0000 7f5f51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.685+0000 7f5f63889640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.685+0000 7f5f63889640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:42.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.685+0000 7f5f63889640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:23:42.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.685+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f3c005330 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.686+0000 7f5f51ffb640 1 -- 172.21.15.19:0/4034866954 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f5f5801e7b0 con 0x7f5f5c0a9d70 2024-03-29T11:23:42.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.686+0000 7f5f51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:42.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.686+0000 7f5f51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:42.667 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:42.667 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:42.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 10 librados: watch_flush enter 2024-03-29T11:23:42.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 10 librados: watch_flush exit 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 >> v1:172.21.15.19:6816/3721232097 conn(0x7f5f4006e000 legacy=0x7f5f400704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 10 monclient: shutdown 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.690+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 >> v1:172.21.15.19:6790/0 conn(0x7f5f5c0a9d70 legacy=0x7f5f5c136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.691+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 shutdown_connections 2024-03-29T11:23:42.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.691+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 >> 172.21.15.19:0/4034866954 conn(0x7f5f5c066a00 msgr2=0x7f5f5c13b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:42.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.691+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 shutdown_connections 2024-03-29T11:23:42.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.691+0000 7f5f63889640 1 -- 172.21.15.19:0/4034866954 wait complete. 2024-03-29T11:23:42.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:42.692+0000 7f5f63889640 1 librados: shutdown 2024-03-29T11:23:42.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:49.496 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:49.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.665+0000 7fc3cbcfb640 10 monclient: get_monmap_and_config 2024-03-29T11:23:49.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.665+0000 7fc3cbcfb640 10 monclient: build_initial_monmap 2024-03-29T11:23:49.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.665+0000 7fc3cbcfb640 10 monclient: monmap: 2024-03-29T11:23:49.645 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:49.666051+0000 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:49.666051+0000 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:49.646 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.666+0000 7fc3cbcfb640 1 Processor -- start 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.666+0000 7fc3cbcfb640 1 -- start start 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.666+0000 7fc3cbcfb640 10 monclient: init 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c406c250 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c4072400 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c406c630 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c406d760 con 0x7fc3c406c250 2024-03-29T11:23:49.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c4070500 con 0x7fc3c406c630 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c40aff50 con 0x7fc3c4072400 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3cbcfb640 10 monclient(hunting): authenticate will time out at 1149.930684s 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3c9a97640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc3c406c250 0x7fc3c4071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48452/0 (socket says 172.21.15.19:48452) 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.667+0000 7fc3c9a97640 1 -- 172.21.15.19:0/1389118302 learned_addr learned my addr 172.21.15.19:0/1389118302 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.668+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494049 0 0) 0x7fc3c4070500 con 0x7fc3c406c630 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.668+0000 7fc3c8a95640 10 monclient(hunting): _negotiate 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.668+0000 7fc3c8a95640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.668+0000 7fc3c8a95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.668+0000 7fc3c8a95640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.668+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b0003520 con 0x7fc3c406c630 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1448200082 0 0) 0x7fc3b0003520 con 0x7fc3c406c630 2024-03-29T11:23:49.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 >> v1:172.21.15.19:6790/0 conn(0x7fc3c4072400 legacy=0x7fc3c41465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 >> v1:172.21.15.19:6789/0 conn(0x7fc3c406c250 legacy=0x7fc3c4071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: _finish_hunting 0 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 1 monclient: found mon.noname-c 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c40b00e0 con 0x7fc3c406c630 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: _finish_auth 0 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: _check_auth_tickets 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3cbcfb640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3cbcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3bc001a10 con 0x7fc3c406c630 2024-03-29T11:23:49.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3c8a95640 10 monclient: dump: 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.652 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:49.653 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.653 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:49.653 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:49.653 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:49.653 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.669+0000 7fc3cbcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:49.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.670+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3bc001bf0 con 0x7fc3c406c630 2024-03-29T11:23:49.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.670+0000 7fc3c8a95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:49.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.670+0000 7fc3cbcfb640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:49.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.670+0000 7fc3cbcfb640 4 set_mon_vals no callback set 2024-03-29T11:23:49.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.670+0000 7fc3c8a95640 1 -- 172.21.15.19:0/1389118302 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3bc001a10 con 0x7fc3c406c630 2024-03-29T11:23:49.665 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.670+0000 7fc3c8a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3c8a95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3c8a95640 10 monclient: dump: 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:49.666 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 10 monclient: shutdown 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1389118302 >> v1:172.21.15.62:6789/0 conn(0x7fc3c406c630 legacy=0x7fc3c4151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1389118302 shutdown_connections 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1389118302 >> 172.21.15.19:0/1389118302 conn(0x7fc3c4067610 msgr2=0x7fc3c40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1389118302 shutdown_connections 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.671+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1389118302 wait complete. 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 10 monclient: build_initial_monmap 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 10 monclient: monmap: 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:49.667 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 1 librados: starting msgr at 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 1 librados: starting objecter 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 1 Processor -- start 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 1 -- start start 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 1 librados: setting wanted keys 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 1 librados: calling monclient init 2024-03-29T11:23:49.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.672+0000 7fc3cbcfb640 10 monclient: init 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3c406c250 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3c406c630 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3c4072400 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c40a9d90 con 0x7fc3c406c250 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c4146250 con 0x7fc3c406c630 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3c41463e0 con 0x7fc3c4072400 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3ca298640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fc3c4072400 0x7fc3c41decf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48232/0 (socket says 172.21.15.19:48232) 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3ca298640 1 -- 172.21.15.19:0/1371204451 learned_addr learned my addr 172.21.15.19:0/1371204451 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.673+0000 7fc3cbcfb640 10 monclient(hunting): authenticate will time out at 1149.936872s 2024-03-29T11:23:49.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 client.?.objecter ms_handle_connect 0x7fc3c406c630 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 client.?.objecter ms_handle_connect 0x7fc3c4072400 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 client.?.objecter ms_handle_connect 0x7fc3c406c250 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758390028 0 0) 0x7fc3c4146250 con 0x7fc3c406c630 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a8004eb0 con 0x7fc3c406c630 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151991974 0 0) 0x7fc3c41463e0 con 0x7fc3c4072400 2024-03-29T11:23:49.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.674+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c4146250 con 0x7fc3c4072400 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223685104 0 0) 0x7fc3c40a9d90 con 0x7fc3c406c250 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c41463e0 con 0x7fc3c406c250 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2364659590 0 0) 0x7fc3a8004eb0 con 0x7fc3c406c630 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c40a9d90 con 0x7fc3c406c630 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3b4001890 con 0x7fc3c406c630 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:49.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.672 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3981840073 0 0) 0x7fc3c4146250 con 0x7fc3c4072400 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3a8004eb0 con 0x7fc3c4072400 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3bc0020e0 con 0x7fc3c4072400 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient(hunting): dump: 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:49.673 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 267195703 0 0) 0x7fc3c40a9d90 con 0x7fc3c406c630 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 >> v1:172.21.15.19:6790/0 conn(0x7fc3c4072400 legacy=0x7fc3c41decf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 >> v1:172.21.15.19:6789/0 conn(0x7fc3c406c250 legacy=0x7fc3c40a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 monclient: found mon.b 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c4146570 con 0x7fc3c406c630 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:49.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3cbcfb640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3cbcfb640 10 monclient: _renew_subs 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.675+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c40b4b90 con 0x7fc3c406c630 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.676+0000 7fc3cbcfb640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.676+0000 7fc3cbcfb640 10 monclient: _renew_subs 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.676+0000 7fc3cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.676+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c413ed60 con 0x7fc3c406c630 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.676+0000 7fc3cbcfb640 1 librados: init done 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.677+0000 7fc3cbcfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.677+0000 7fc3cbcfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:49.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.677+0000 7fc3cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.677+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c4075f10 con 0x7fc3c406c630 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.677+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3b40019f0 con 0x7fc3c406c630 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.677+0000 7fc3ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.678+0000 7fc3bbfff640 4 set_mon_vals no callback set 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.679+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3b4003790 con 0x7fc3c406c630 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.679+0000 7fc3ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.679+0000 7fc3ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.679+0000 7fc3ba7fc640 10 monclient: dump: 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:49.676 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.679+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fc3b40039b0 con 0x7fc3c406c630 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.680+0000 7fc3c9a97640 10 monclient: get_auth_request con 0x7fc3a806df70 auth_method 0 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.680+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fc3b4083880 con 0x7fc3c406c630 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.680+0000 7fc3ba7fc640 10 client.4384.objecter ms_dispatch 0x7fc3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.680+0000 7fc3ba7fc640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:49.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.680+0000 7fc3ba7fc640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:49.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.680+0000 7fc3ba7fc640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-29T11:23:49.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.681+0000 7fc3ba7fc640 10 client.4384.objecter ms_handle_connect 0x7fc3a806df70 2024-03-29T11:23:49.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.685+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3b404f1e0 con 0x7fc3c406c630 2024-03-29T11:23:49.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.685+0000 7fc3ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:49.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:49.685+0000 7fc3ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:50.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.057+0000 7fc3cbcfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:50.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.058+0000 7fc3cbcfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:50.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.058+0000 7fc3cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:23:50.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.058+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3c40b4d20 con 0x7fc3c406c630 2024-03-29T11:23:50.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.059+0000 7fc3ba7fc640 1 -- 172.21.15.19:0/1371204451 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fc3b4051110 con 0x7fc3c406c630 2024-03-29T11:23:50.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.059+0000 7fc3ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:50.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.059+0000 7fc3ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:50.039 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:50.039 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.062+0000 7fc3cbcfb640 10 librados: watch_flush enter 2024-03-29T11:23:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 10 librados: watch_flush exit 2024-03-29T11:23:50.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-29T11:23:50.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 10 client.4384.objecter successfully canceled tick 2024-03-29T11:23:50.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 >> v1:172.21.15.19:6816/3721232097 conn(0x7fc3a806df70 legacy=0x7fc3a8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 10 monclient: shutdown 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 >> v1:172.21.15.62:6789/0 conn(0x7fc3c406c630 legacy=0x7fc3c413e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 shutdown_connections 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 >> 172.21.15.19:0/1371204451 conn(0x7fc3c4067610 msgr2=0x7fc3c4149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 shutdown_connections 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.063+0000 7fc3cbcfb640 1 -- 172.21.15.19:0/1371204451 wait complete. 2024-03-29T11:23:50.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:50.064+0000 7fc3cbcfb640 1 librados: shutdown 2024-03-29T11:23:50.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:56.867 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:57.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.044+0000 7fd1aa7cf640 10 monclient: get_monmap_and_config 2024-03-29T11:23:57.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.044+0000 7fd1aa7cf640 10 monclient: build_initial_monmap 2024-03-29T11:23:57.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.044+0000 7fd1aa7cf640 10 monclient: monmap: 2024-03-29T11:23:57.025 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:57.025 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:23:57.045146+0000 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:23:57.045146+0000 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:57.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.045+0000 7fd1aa7cf640 1 Processor -- start 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.046+0000 7fd1aa7cf640 1 -- start start 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.046+0000 7fd1aa7cf640 10 monclient: init 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.046+0000 7fd1aa7cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.046+0000 7fd1aa7cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a40a9ac0 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a413ee90 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a40af900 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:57.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40afd10 con 0x7fd1a413ee90 2024-03-29T11:23:57.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40af6f0 con 0x7fd1a40af900 2024-03-29T11:23:57.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40aac80 con 0x7fd1a40a9ac0 2024-03-29T11:23:57.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:57.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1aa7cf640 10 monclient(hunting): authenticate will time out at 1157.310081s 2024-03-29T11:23:57.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1a8d6c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd1a40af900 0x7fd1a4149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:43402/0 (socket says 172.21.15.19:43402) 2024-03-29T11:23:57.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.047+0000 7fd1a8d6c640 1 -- 172.21.15.19:0/2171436869 learned_addr learned my addr 172.21.15.19:0/2171436869 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352352819 0 0) 0x7fd1a40af6f0 con 0x7fd1a40af900 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd18c003520 con 0x7fd1a40af900 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1591629656 0 0) 0x7fd18c003520 con 0x7fd1a40af900 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.048+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 >> v1:172.21.15.19:6790/0 conn(0x7fd1a40a9ac0 legacy=0x7fd1a40adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 >> v1:172.21.15.19:6789/0 conn(0x7fd1a413ee90 legacy=0x7fd1a413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:57.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40aaf60 con 0x7fd1a40af900 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd194001690 con 0x7fd1a40af900 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1a2ffd640 10 monclient: dump: 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:57.032 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1aa7cf640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.049+0000 7fd1aa7cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd194001920 con 0x7fd1a40af900 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1a2ffd640 1 -- 172.21.15.19:0/2171436869 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd194001d30 con 0x7fd1a40af900 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1a2ffd640 10 monclient: dump: 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:57.033 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1aa7cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:57.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.050+0000 7fd1aa7cf640 4 set_mon_vals no callback set 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.051+0000 7fd1aa7cf640 10 monclient: shutdown 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.051+0000 7fd1aa7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.051+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/2171436869 >> v1:172.21.15.62:6789/0 conn(0x7fd1a40af900 legacy=0x7fd1a4149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.051+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/2171436869 shutdown_connections 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.051+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/2171436869 >> 172.21.15.19:0/2171436869 conn(0x7fd1a40b35e0 msgr2=0x7fd1a40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.051+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/2171436869 shutdown_connections 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.052+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/2171436869 wait complete. 2024-03-29T11:23:57.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 10 monclient: build_initial_monmap 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 10 monclient: monmap: 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:57.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 1 librados: starting msgr at 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 1 librados: starting objecter 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 1 Processor -- start 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 1 -- start start 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 1 librados: setting wanted keys 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 1 librados: calling monclient init 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.053+0000 7fd1aa7cf640 10 monclient: init 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd1a40a9ac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd1a40af900 addr v1:172.21.15.19:6789/0 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd1a413ee90 addr v1:172.21.15.19:6790/0 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a41418c0 con 0x7fd1a40af900 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a4141ba0 con 0x7fd1a40a9ac0 2024-03-29T11:23:57.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a412aad0 con 0x7fd1a413ee90 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.054+0000 7fd1aa7cf640 10 monclient(hunting): authenticate will time out at 1157.317810s 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd1a40a9ac0 0x7fd1a41402c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:43758/0 (socket says 172.21.15.19:43758) 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a3fff640 1 -- 172.21.15.19:0/3779792832 learned_addr learned my addr 172.21.15.19:0/3779792832 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd1a40a9ac0 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd1a40af900 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 client.?.objecter ms_handle_connect 0x7fd1a413ee90 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068995079 0 0) 0x7fd1a41418c0 con 0x7fd1a40af900 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:57.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190004eb0 con 0x7fd1a40af900 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231305024 0 0) 0x7fd1a412aad0 con 0x7fd1a413ee90 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.055+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a41418c0 con 0x7fd1a413ee90 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604349446 0 0) 0x7fd1a4141ba0 con 0x7fd1a40a9ac0 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:57.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a412aad0 con 0x7fd1a40a9ac0 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198330202 0 0) 0x7fd190004eb0 con 0x7fd1a40af900 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1a4141ba0 con 0x7fd1a40af900 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd19c001b50 con 0x7fd1a40af900 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:57.043 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 287740805 0 0) 0x7fd1a4141ba0 con 0x7fd1a40af900 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 >> v1:172.21.15.19:6790/0 conn(0x7fd1a413ee90 legacy=0x7fd1a41df480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 >> v1:172.21.15.62:6789/0 conn(0x7fd1a40a9ac0 legacy=0x7fd1a41402c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 monclient: found mon.a 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:57.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a412ac60 con 0x7fd1a40af900 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1aa7cf640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1aa7cf640 10 monclient: _renew_subs 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1aa7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.056+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a41422a0 con 0x7fd1a40af900 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1aa7cf640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1aa7cf640 10 monclient: _renew_subs 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1aa7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a406e980 con 0x7fd1a40af900 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1aa7cf640 1 librados: init done 2024-03-29T11:23:57.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd19c002060 con 0x7fd1a40af900 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a27fc640 4 set_mon_vals no callback set 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd19c003b30 con 0x7fd1a40af900 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.057+0000 7fd1a0ff9640 10 monclient: dump: 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:23:57.046 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:23:57.047 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:23:57.047 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.059+0000 7fd1aa7cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.059+0000 7fd1aa7cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.059+0000 7fd1aa7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.059+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd184005080 con 0x7fd1a40af900 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.059+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fd19c003c90 con 0x7fd1a40af900 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fd19c083b70 con 0x7fd1a40af900 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a0ff9640 10 client.4398.objecter ms_dispatch 0x7fd1a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a0ff9640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a0ff9640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a0ff9640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a3fff640 10 monclient: get_auth_request con 0x7fd19006dd80 auth_method 0 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.060+0000 7fd1a0ff9640 10 client.4398.objecter ms_handle_connect 0x7fd19006dd80 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.067+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd19c04f280 con 0x7fd1a40af900 2024-03-29T11:23:57.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.068+0000 7fd1a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:57.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.068+0000 7fd1a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:57.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.444+0000 7fd1aa7cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:57.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.444+0000 7fd1aa7cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:57.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.444+0000 7fd1aa7cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:23:57.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.444+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd184005330 con 0x7fd1a40af900 2024-03-29T11:23:57.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.445+0000 7fd1a0ff9640 1 -- 172.21.15.19:0/3779792832 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fd19c051380 con 0x7fd1a40af900 2024-03-29T11:23:57.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.445+0000 7fd1a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:57.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.445+0000 7fd1a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:57.425 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:23:57.426 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.448+0000 7fd1aa7cf640 10 librados: watch_flush enter 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 10 librados: watch_flush exit 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 >> v1:172.21.15.19:6816/3721232097 conn(0x7fd19006dd80 legacy=0x7fd190070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 10 monclient: shutdown 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 >> v1:172.21.15.19:6789/0 conn(0x7fd1a40af900 legacy=0x7fd1a414c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:57.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 shutdown_connections 2024-03-29T11:23:57.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 >> 172.21.15.19:0/3779792832 conn(0x7fd1a40b35e0 msgr2=0x7fd1a40ad040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:57.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 shutdown_connections 2024-03-29T11:23:57.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 -- 172.21.15.19:0/3779792832 wait complete. 2024-03-29T11:23:57.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:23:57.449+0000 7fd1aa7cf640 1 librados: shutdown 2024-03-29T11:23:57.445 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:04.252 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:04.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.424+0000 7f8af59fa640 10 monclient: get_monmap_and_config 2024-03-29T11:24:04.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.424+0000 7f8af59fa640 10 monclient: build_initial_monmap 2024-03-29T11:24:04.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.424+0000 7f8af59fa640 10 monclient: monmap: 2024-03-29T11:24:04.405 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:04.425334+0000 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:04.425334+0000 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:04.406 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:04.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.425+0000 7f8af59fa640 1 Processor -- start 2024-03-29T11:24:04.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.425+0000 7f8af59fa640 1 -- start start 2024-03-29T11:24:04.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.425+0000 7f8af59fa640 10 monclient: init 2024-03-29T11:24:04.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8ae802b570 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae8006800 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae8006c80 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae8032ef0 con 0x7f8ae8006c80 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae80be170 con 0x7f8ae802b570 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae802c720 con 0x7f8ae8006800 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:04.408 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.426+0000 7f8af59fa640 10 monclient(hunting): authenticate will time out at 1164.689747s 2024-03-29T11:24:04.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.429+0000 7f8af49f8640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8ae802b570 0x7f8ae80345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:40205/0 (socket says 172.21.15.19:40205) 2024-03-29T11:24:04.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.429+0000 7f8af49f8640 1 -- 172.21.15.19:0/2297458315 learned_addr learned my addr 172.21.15.19:0/2297458315 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:04.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581287028 0 0) 0x7f8ae80be170 con 0x7f8ae802b570 2024-03-29T11:24:04.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:04.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:04.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:04.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae0005730 con 0x7f8ae802b570 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135382820 0 0) 0x7f8ae802c720 con 0x7f8ae8006800 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae80be170 con 0x7f8ae8006800 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851917166 0 0) 0x7f8af004fdd0 con 0x7f8ae8006c80 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.430+0000 7f8aef7fe640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ae0006c40 con 0x7f8ae8006c80 2024-03-29T11:24:04.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835677812 0 0) 0x7f8ae0005730 con 0x7f8ae802b570 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 >> v1:172.21.15.19:6790/0 conn(0x7f8ae8006800 legacy=0x7f8ae80bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 >> v1:172.21.15.19:6789/0 conn(0x7f8ae8006c80 legacy=0x7f8ae80c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 monclient: found mon.noname-c 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae802c990 con 0x7f8ae802b570 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8adc001690 con 0x7f8ae802b570 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:04.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: dump: 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8af59fa640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8af59fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8adc0017f0 con 0x7f8ae802b570 2024-03-29T11:24:04.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 1 -- 172.21.15.19:0/2297458315 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8adc001c00 con 0x7f8ae802b570 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8aef7fe640 10 monclient: dump: 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:04.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8af59fa640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.431+0000 7f8af59fa640 4 set_mon_vals no callback set 2024-03-29T11:24:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.432+0000 7f8af59fa640 10 monclient: shutdown 2024-03-29T11:24:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.432+0000 7f8af59fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.432+0000 7f8af59fa640 1 -- 172.21.15.19:0/2297458315 >> v1:172.21.15.62:6789/0 conn(0x7f8ae802b570 legacy=0x7f8ae80345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.432+0000 7f8af59fa640 1 -- 172.21.15.19:0/2297458315 shutdown_connections 2024-03-29T11:24:04.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.432+0000 7f8af59fa640 1 -- 172.21.15.19:0/2297458315 >> 172.21.15.19:0/2297458315 conn(0x7f8ae80058f0 msgr2=0x7f8ae8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.432+0000 7f8af59fa640 1 -- 172.21.15.19:0/2297458315 shutdown_connections 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.433+0000 7f8af59fa640 1 -- 172.21.15.19:0/2297458315 wait complete. 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.434+0000 7f8af59fa640 10 monclient: build_initial_monmap 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.434+0000 7f8af59fa640 10 monclient: monmap: 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:04.421 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 1 librados: starting msgr at 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 1 librados: starting objecter 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 1 Processor -- start 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 1 -- start start 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 1 librados: setting wanted keys 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 1 librados: calling monclient init 2024-03-29T11:24:04.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.435+0000 7f8af59fa640 10 monclient: init 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.436+0000 7f8af59fa640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.436+0000 7f8af59fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.436+0000 7f8af59fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae8006800 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.436+0000 7f8af59fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae8006c80 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.437+0000 7f8af59fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8ae802b570 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.437+0000 7f8af59fa640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae8030cf0 con 0x7f8ae8006c80 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.437+0000 7f8af59fa640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae8031180 con 0x7f8ae8006800 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.437+0000 7f8af59fa640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae80c26d0 con 0x7f8ae802b570 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.437+0000 7f8af59fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.437+0000 7f8af59fa640 10 monclient(hunting): authenticate will time out at 1164.700825s 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8af51f9640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f8ae802b570 0x7f8ae815d8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:45412/0 (socket says 172.21.15.19:45412) 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8af51f9640 1 -- 172.21.15.19:0/3868909135 learned_addr learned my addr 172.21.15.19:0/3868909135 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 client.?.objecter ms_handle_connect 0x7f8ae802b570 2024-03-29T11:24:04.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902657785 0 0) 0x7f8ae80c26d0 con 0x7f8ae802b570 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad4004eb0 con 0x7f8ae802b570 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3449448957 0 0) 0x7f8ad4004eb0 con 0x7f8ae802b570 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ae80c26d0 con 0x7f8ae802b570 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8af0068890 con 0x7f8ae802b570 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): dump: 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:04.424 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.438+0000 7f8aed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.439+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940757048 0 0) 0x7f8ae80c26d0 con 0x7f8ae802b570 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.439+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 >> v1:172.21.15.62:6789/0 conn(0x7f8ae8006800 legacy=0x7f8ae8169a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.439+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 >> v1:172.21.15.19:6789/0 conn(0x7f8ae8006c80 legacy=0x7f8ae815d1c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 1 monclient: found mon.c 2024-03-29T11:24:04.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae80c1ae0 con 0x7f8ae802b570 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8af004fdd0 con 0x7f8ae802b570 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:04.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8af004f730 con 0x7f8ae802b570 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8aed7fa640 10 monclient: dump: 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:04.429 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 10 monclient: _renew_subs 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae815e040 con 0x7f8ae802b570 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 10 monclient: _renew_subs 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae8030840 con 0x7f8ae802b570 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.440+0000 7f8af59fa640 1 librados: init done 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.441+0000 7f8aeeffd640 4 set_mon_vals no callback set 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.443+0000 7f8af59fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:04.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.443+0000 7f8af59fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.443+0000 7f8af59fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.443+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad0005080 con 0x7f8ae802b570 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.443+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f8af0075710 con 0x7f8ae802b570 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.444+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f8af00ff5c0 con 0x7f8ae802b570 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.444+0000 7f8aed7fa640 10 client.4421.objecter ms_dispatch 0x7f8af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.444+0000 7f8aed7fa640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.444+0000 7f8aed7fa640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.444+0000 7f8aed7fa640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.447+0000 7f8af49f8640 10 monclient: get_auth_request con 0x7f8ad406daf0 auth_method 0 2024-03-29T11:24:04.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.448+0000 7f8aed7fa640 10 client.4421.objecter ms_handle_connect 0x7f8ad406daf0 2024-03-29T11:24:04.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.455+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8af00caea0 con 0x7f8ae802b570 2024-03-29T11:24:04.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.455+0000 7f8aed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:04.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.455+0000 7f8aed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:04.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.788+0000 7f8af59fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:04.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.788+0000 7f8af59fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:04.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.788+0000 7f8af59fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:04.769 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.788+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ad0005330 con 0x7f8ae802b570 2024-03-29T11:24:04.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.792+0000 7f8aed7fa640 1 -- 172.21.15.19:0/3868909135 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f8af00ccdd0 con 0x7f8ae802b570 2024-03-29T11:24:04.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.792+0000 7f8aed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:04.773 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.792+0000 7f8aed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:04.774 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:04.774 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:04.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.796+0000 7f8af59fa640 10 librados: watch_flush enter 2024-03-29T11:24:04.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.796+0000 7f8af59fa640 10 librados: watch_flush exit 2024-03-29T11:24:04.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.796+0000 7f8af59fa640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:24:04.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.796+0000 7f8af59fa640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:24:04.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.799+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 >> v1:172.21.15.19:6816/3721232097 conn(0x7f8ad406daf0 legacy=0x7f8ad406ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.799+0000 7f8af59fa640 10 monclient: shutdown 2024-03-29T11:24:04.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.799+0000 7f8af59fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:04.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.799+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 >> v1:172.21.15.19:6790/0 conn(0x7f8ae802b570 legacy=0x7f8ae815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:04.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.800+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 shutdown_connections 2024-03-29T11:24:04.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.800+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 >> 172.21.15.19:0/3868909135 conn(0x7f8ae80058f0 msgr2=0x7f8ae8034460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:04.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.800+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 shutdown_connections 2024-03-29T11:24:04.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.800+0000 7f8af59fa640 1 -- 172.21.15.19:0/3868909135 wait complete. 2024-03-29T11:24:04.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:04.800+0000 7f8af59fa640 1 librados: shutdown 2024-03-29T11:24:04.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:11.605 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:11.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.776+0000 7f63caf63640 10 monclient: get_monmap_and_config 2024-03-29T11:24:11.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.776+0000 7f63caf63640 10 monclient: build_initial_monmap 2024-03-29T11:24:11.757 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.776+0000 7f63caf63640 10 monclient: monmap: 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:11.777247+0000 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:11.777247+0000 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:11.758 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:11.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.777+0000 7f63caf63640 1 Processor -- start 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 1 -- start start 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient: init 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient(hunting): picked mon.noname-c con 0x7f63c40b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient(hunting): picked mon.noname-b con 0x7f63c414a700 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient(hunting): picked mon.noname-a con 0x7f63c40ac5a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:11.764 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40aba70 con 0x7f63c40ac5a0 2024-03-29T11:24:11.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40aecb0 con 0x7f63c40b0770 2024-03-29T11:24:11.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40af150 con 0x7f63c414a700 2024-03-29T11:24:11.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:11.765 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.778+0000 7f63caf63640 10 monclient(hunting): authenticate will time out at 1172.041967s 2024-03-29T11:24:11.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.785+0000 7f63c8cff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f63c40b0770 0x7f63c4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:40347/0 (socket says 172.21.15.19:40347) 2024-03-29T11:24:11.766 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.785+0000 7f63c8cff640 1 -- 172.21.15.19:0/1725612478 learned_addr learned my addr 172.21.15.19:0/1725612478 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.786+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516758045 0 0) 0x7f63c40aba70 con 0x7f63c40ac5a0 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003520 con 0x7f63c40ac5a0 2024-03-29T11:24:11.776 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569448853 0 0) 0x7f63c40af150 con 0x7f63c414a700 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c40aba70 con 0x7f63c414a700 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 117104400 0 0) 0x7f63ac003520 con 0x7f63c40ac5a0 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 >> v1:172.21.15.19:6790/0 conn(0x7f63c414a700 legacy=0x7f63c40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 >> v1:172.21.15.62:6789/0 conn(0x7f63c40b0770 legacy=0x7f63c4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:24:11.777 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c40ac9d0 con 0x7f63c40ac5a0 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f63b4001690 con 0x7f63c40ac5a0 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63c37fe640 10 monclient: dump: 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:11.778 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63caf63640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.787+0000 7f63caf63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63b4001870 con 0x7f63c40ac5a0 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63caf63640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63caf63640 4 set_mon_vals no callback set 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63c37fe640 1 -- 172.21.15.19:0/1725612478 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f63b4001c40 con 0x7f63c40ac5a0 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63c37fe640 10 monclient: dump: 2024-03-29T11:24:11.779 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63caf63640 10 monclient: shutdown 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63caf63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.788+0000 7f63caf63640 1 -- 172.21.15.19:0/1725612478 >> v1:172.21.15.19:6789/0 conn(0x7f63c40ac5a0 legacy=0x7f63c4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.789+0000 7f63caf63640 1 -- 172.21.15.19:0/1725612478 shutdown_connections 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.789+0000 7f63caf63640 1 -- 172.21.15.19:0/1725612478 >> 172.21.15.19:0/1725612478 conn(0x7f63c406b660 msgr2=0x7f63c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:11.780 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.789+0000 7f63caf63640 1 -- 172.21.15.19:0/1725612478 shutdown_connections 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.789+0000 7f63caf63640 1 -- 172.21.15.19:0/1725612478 wait complete. 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 10 monclient: build_initial_monmap 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 10 monclient: monmap: 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:11.781 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 1 librados: starting msgr at 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 1 librados: starting objecter 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 1 Processor -- start 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 1 -- start start 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 1 librados: setting wanted keys 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 1 librados: calling monclient init 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.790+0000 7f63caf63640 10 monclient: init 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient(hunting): picked mon.noname-c con 0x7f63c40ac5a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient(hunting): picked mon.noname-b con 0x7f63c40b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient(hunting): picked mon.noname-a con 0x7f63c414a700 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:11.782 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c41525e0 con 0x7f63c414a700 2024-03-29T11:24:11.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c41377c0 con 0x7f63c40b0770 2024-03-29T11:24:11.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c4150ee0 con 0x7f63c40ac5a0 2024-03-29T11:24:11.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:11.783 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.791+0000 7f63caf63640 10 monclient(hunting): authenticate will time out at 1172.054844s 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.795+0000 7f63c3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f63c40b0770 0x7f63c41370b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:40427/0 (socket says 172.21.15.19:40427) 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.795+0000 7f63c3fff640 1 -- 172.21.15.19:0/1813518119 learned_addr learned my addr 172.21.15.19:0/1813518119 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.796+0000 7f63c17fa640 10 client.?.objecter ms_handle_connect 0x7f63c40b0770 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.796+0000 7f63c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325622388 0 0) 0x7f63c41377c0 con 0x7f63c40b0770 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:24:11.788 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b0004eb0 con 0x7f63c40b0770 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2631622670 0 0) 0x7f63b0004eb0 con 0x7f63c40b0770 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.798+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b0005300 con 0x7f63c40b0770 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f63bc001c50 con 0x7f63c40b0770 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient(hunting): dump: 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:11.789 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 client.?.objecter ms_handle_connect 0x7f63c40ac5a0 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1999223255 0 0) 0x7f63b0005300 con 0x7f63c40b0770 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 >> v1:172.21.15.19:6790/0 conn(0x7f63c40ac5a0 legacy=0x7f63c414ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 >> v1:172.21.15.19:6789/0 conn(0x7f63c414a700 legacy=0x7f63c41ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 1 monclient: found mon.b 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:11.790 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c41528c0 con 0x7f63c40b0770 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 10 monclient: _renew_subs 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c4151180 con 0x7f63c40b0770 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 10 monclient: _renew_subs 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4150920 con 0x7f63c40b0770 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.799+0000 7f63caf63640 1 librados: init done 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63bc003970 con 0x7f63c40b0770 2024-03-29T11:24:11.791 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f63bc001c50 con 0x7f63c40b0770 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c17fa640 10 monclient: dump: 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:11.792 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.800+0000 7f63c2ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.801+0000 7f63caf63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.801+0000 7f63caf63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.801+0000 7f63caf63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.801+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a4005080 con 0x7f63c40b0770 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.802+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f63bc003be0 con 0x7f63c40b0770 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.802+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f63bc084960 con 0x7f63c40b0770 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.802+0000 7f63c17fa640 10 client.4399.objecter ms_dispatch 0x7f63c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.802+0000 7f63c17fa640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.802+0000 7f63c17fa640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.802+0000 7f63c17fa640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.803+0000 7f63c8cff640 10 monclient: get_auth_request con 0x7f63b006e1c0 auth_method 0 2024-03-29T11:24:11.793 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.803+0000 7f63c17fa640 10 client.4399.objecter ms_handle_connect 0x7f63b006e1c0 2024-03-29T11:24:11.794 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.809+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63bc050070 con 0x7f63c40b0770 2024-03-29T11:24:11.794 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.809+0000 7f63c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:11.794 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:11.809+0000 7f63c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:12.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.166+0000 7f63caf63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.166+0000 7f63caf63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.166+0000 7f63caf63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.166+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63a4005330 con 0x7f63c40b0770 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.169+0000 7f63c17fa640 1 -- 172.21.15.19:0/1813518119 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f63bc001ab0 con 0x7f63c40b0770 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.169+0000 7f63c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.169+0000 7f63c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:12.150 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:12.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.172+0000 7f63caf63640 10 librados: watch_flush enter 2024-03-29T11:24:12.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.174+0000 7f63caf63640 10 librados: watch_flush exit 2024-03-29T11:24:12.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.174+0000 7f63caf63640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T11:24:12.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.174+0000 7f63caf63640 10 client.4399.objecter successfully canceled tick 2024-03-29T11:24:12.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.175+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 >> v1:172.21.15.19:6816/3721232097 conn(0x7f63b006e1c0 legacy=0x7f63b0070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:12.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.175+0000 7f63caf63640 10 monclient: shutdown 2024-03-29T11:24:12.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.175+0000 7f63caf63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:12.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.175+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 >> v1:172.21.15.62:6789/0 conn(0x7f63c40b0770 legacy=0x7f63c41370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:12.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.175+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 shutdown_connections 2024-03-29T11:24:12.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.175+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 >> 172.21.15.19:0/1813518119 conn(0x7f63c406b660 msgr2=0x7f63c406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:12.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.178+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 shutdown_connections 2024-03-29T11:24:12.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.178+0000 7f63caf63640 1 -- 172.21.15.19:0/1813518119 wait complete. 2024-03-29T11:24:12.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:12.179+0000 7f63caf63640 1 librados: shutdown 2024-03-29T11:24:12.178 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:18.985 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.152+0000 7fa445fd7640 10 monclient: get_monmap_and_config 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.152+0000 7fa445fd7640 10 monclient: build_initial_monmap 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.152+0000 7fa445fd7640 10 monclient: monmap: 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:19.153596+0000 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:19.153596+0000 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:19.135 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:19.136 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:19.136 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:19.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.154+0000 7fa445fd7640 1 Processor -- start 2024-03-29T11:24:19.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.154+0000 7fa445fd7640 1 -- start start 2024-03-29T11:24:19.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.154+0000 7fa445fd7640 10 monclient: init 2024-03-29T11:24:19.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:19.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:19.137 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient(hunting): picked mon.noname-c con 0x7fa4400a9ac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient(hunting): picked mon.noname-b con 0x7fa44013ee90 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient(hunting): picked mon.noname-a con 0x7fa4400af900 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400afd10 con 0x7fa4400af900 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400af6f0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4400aac80 con 0x7fa44013ee90 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:19.138 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.155+0000 7fa445fd7640 10 monclient(hunting): authenticate will time out at 1179.418731s 2024-03-29T11:24:19.139 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.159+0000 7fa43f7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fa4400a9ac0 0x7fa4400adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:53787/0 (socket says 172.21.15.19:53787) 2024-03-29T11:24:19.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.161+0000 7fa43effd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa44013ee90 0x7fa44013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:37227/0 (socket says 172.21.15.19:37227) 2024-03-29T11:24:19.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.161+0000 7fa43effd640 1 -- 172.21.15.19:0/160890831 learned_addr learned my addr 172.21.15.19:0/160890831 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:19.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845592283 0 0) 0x7fa4400afd10 con 0x7fa4400af900 2024-03-29T11:24:19.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:19.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa428003520 con 0x7fa4400af900 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4141528769 0 0) 0x7fa428003520 con 0x7fa4400af900 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 >> v1:172.21.15.19:6790/0 conn(0x7fa44013ee90 legacy=0x7fa44013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 >> v1:172.21.15.62:6789/0 conn(0x7fa4400a9ac0 legacy=0x7fa4400adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:24:19.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4400aaf60 con 0x7fa4400af900 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa434001690 con 0x7fa4400af900 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.162+0000 7fa43e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 10 monclient: dump: 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.150 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa445fd7640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa445fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa434001860 con 0x7fa4400af900 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 1 -- 172.21.15.19:0/160890831 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa434001cb0 con 0x7fa4400af900 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:19.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa43e7fc640 10 monclient: dump: 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa445fd7640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.163+0000 7fa445fd7640 4 set_mon_vals no callback set 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.164+0000 7fa445fd7640 10 monclient: shutdown 2024-03-29T11:24:19.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.164+0000 7fa445fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:19.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.164+0000 7fa445fd7640 1 -- 172.21.15.19:0/160890831 >> v1:172.21.15.19:6789/0 conn(0x7fa4400af900 legacy=0x7fa440149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.164+0000 7fa445fd7640 1 -- 172.21.15.19:0/160890831 shutdown_connections 2024-03-29T11:24:19.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.164+0000 7fa445fd7640 1 -- 172.21.15.19:0/160890831 >> 172.21.15.19:0/160890831 conn(0x7fa4400b35e0 msgr2=0x7fa4400b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:19.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.167+0000 7fa445fd7640 1 -- 172.21.15.19:0/160890831 shutdown_connections 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.168+0000 7fa445fd7640 1 -- 172.21.15.19:0/160890831 wait complete. 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.169+0000 7fa445fd7640 10 monclient: build_initial_monmap 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.169+0000 7fa445fd7640 10 monclient: monmap: 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:19.154 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:19.155 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:19.155 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:19.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 1 librados: starting msgr at 2024-03-29T11:24:19.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 1 librados: starting objecter 2024-03-29T11:24:19.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 1 Processor -- start 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 1 -- start start 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 1 librados: setting wanted keys 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 1 librados: calling monclient init 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.170+0000 7fa445fd7640 10 monclient: init 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.175+0000 7fa445fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.175+0000 7fa445fd7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:19.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.175+0000 7fa445fd7640 10 monclient(hunting): picked mon.noname-a con 0x7fa4400a9ac0 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:19.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.177+0000 7fa445fd7640 10 monclient(hunting): picked mon.noname-c con 0x7fa4400af900 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa445fd7640 10 monclient(hunting): picked mon.noname-b con 0x7fa44013ee90 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa445fd7640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44006f4f0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa445fd7640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa440142380 con 0x7fa44013ee90 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa445fd7640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa44006f720 con 0x7fa4400af900 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa445fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa445fd7640 10 monclient(hunting): authenticate will time out at 1179.441426s 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa43f7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fa4400a9ac0 0x7fa440070300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:38769/0 (socket says 172.21.15.19:38769) 2024-03-29T11:24:19.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.178+0000 7fa43f7fe640 1 -- 172.21.15.19:0/51649048 learned_addr learned my addr 172.21.15.19:0/51649048 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:19.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 client.?.objecter ms_handle_connect 0x7fa4400a9ac0 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048937531 0 0) 0x7fa44006f4f0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa42c004eb0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3052337514 0 0) 0x7fa42c004eb0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa44006f4f0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa438001890 con 0x7fa4400a9ac0 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:19.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): dump: 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.179+0000 7fa427fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.180+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 87581654 0 0) 0x7fa44006f4f0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.180+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 >> v1:172.21.15.19:6790/0 conn(0x7fa4400af900 legacy=0x7fa440070a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.180+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 >> v1:172.21.15.62:6789/0 conn(0x7fa44013ee90 legacy=0x7fa4401e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 1 monclient: found mon.a 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44006fa00 con 0x7fa4400a9ac0 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa427fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 10 monclient: _renew_subs 2024-03-29T11:24:19.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44014f260 con 0x7fa4400a9ac0 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 10 monclient: _renew_subs 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa440142040 con 0x7fa4400a9ac0 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.181+0000 7fa445fd7640 1 librados: init done 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.182+0000 7fa425ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.182+0000 7fa425ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.182+0000 7fa425ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:19.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.182+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa440075f10 con 0x7fa4400a9ac0 2024-03-29T11:24:19.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.188+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa438001ee0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.188+0000 7fa427fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:19.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.188+0000 7fa43dffb640 4 set_mon_vals no callback set 2024-03-29T11:24:19.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa4380050c0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:19.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:19.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 10 monclient: dump: 2024-03-29T11:24:19.171 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fa43801b090 con 0x7fa4400a9ac0 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fa438084fb0 con 0x7fa4400a9ac0 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 10 client.4410.objecter ms_dispatch 0x7fa440000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:19.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa43effd640 10 monclient: get_auth_request con 0x7fa42c06dc20 auth_method 0 2024-03-29T11:24:19.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:19.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.189+0000 7fa427fff640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-29T11:24:19.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.190+0000 7fa427fff640 10 client.4410.objecter ms_handle_connect 0x7fa42c06dc20 2024-03-29T11:24:19.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.196+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa438050910 con 0x7fa4400a9ac0 2024-03-29T11:24:19.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.196+0000 7fa427fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:19.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.196+0000 7fa427fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:19.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.544+0000 7fa425ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:19.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.544+0000 7fa425ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:19.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.544+0000 7fa425ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:19.527 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.544+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4400ae010 con 0x7fa4400a9ac0 2024-03-29T11:24:19.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.547+0000 7fa427fff640 1 -- 172.21.15.19:0/51649048 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fa438052840 con 0x7fa4400a9ac0 2024-03-29T11:24:19.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.547+0000 7fa427fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:19.528 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.547+0000 7fa427fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:19.528 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:19.528 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:19.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.550+0000 7fa425ffb640 10 librados: watch_flush enter 2024-03-29T11:24:19.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.550+0000 7fa425ffb640 10 librados: watch_flush exit 2024-03-29T11:24:19.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.550+0000 7fa425ffb640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-29T11:24:19.532 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.550+0000 7fa425ffb640 10 client.4410.objecter successfully canceled tick 2024-03-29T11:24:19.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.551+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 >> v1:172.21.15.19:6816/3721232097 conn(0x7fa42c06dc20 legacy=0x7fa42c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.551+0000 7fa425ffb640 10 monclient: shutdown 2024-03-29T11:24:19.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.551+0000 7fa425ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:19.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.551+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 >> v1:172.21.15.19:6789/0 conn(0x7fa4400a9ac0 legacy=0x7fa440070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:19.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.556+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 shutdown_connections 2024-03-29T11:24:19.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.556+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 >> 172.21.15.19:0/51649048 conn(0x7fa4400b35e0 msgr2=0x7fa4400acf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:19.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.556+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 shutdown_connections 2024-03-29T11:24:19.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.556+0000 7fa425ffb640 1 -- 172.21.15.19:0/51649048 wait complete. 2024-03-29T11:24:19.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:19.556+0000 7fa425ffb640 1 librados: shutdown 2024-03-29T11:24:19.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:26.364 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.539+0000 7ff689b1a640 10 monclient: get_monmap_and_config 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.539+0000 7ff689b1a640 10 monclient: build_initial_monmap 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.539+0000 7ff689b1a640 10 monclient: monmap: 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:26.540724+0000 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:26.540724+0000 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:26.533 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.541+0000 7ff689b1a640 1 Processor -- start 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.541+0000 7ff689b1a640 1 -- start start 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.541+0000 7ff689b1a640 10 monclient: init 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient(hunting): picked mon.noname-c con 0x7ff6840aaee0 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6840ae460 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient(hunting): picked mon.noname-b con 0x7ff6840aeff0 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6840abd90 con 0x7ff6840ae460 2024-03-29T11:24:26.534 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6840ac070 con 0x7ff6840aaee0 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6840ac350 con 0x7ff6840aeff0 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.542+0000 7ff689b1a640 10 monclient(hunting): authenticate will time out at 1186.805364s 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff689319640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff6840aeff0 0x7ff6840af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:59745/0 (socket says 172.21.15.19:59745) 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff689319640 1 -- 172.21.15.19:0/600983479 learned_addr learned my addr 172.21.15.19:0/600983479 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752712084 0 0) 0x7ff6840ac350 con 0x7ff6840aeff0 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:24:26.535 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff674003520 con 0x7ff6840aeff0 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.546+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659378071 0 0) 0x7ff674003520 con 0x7ff6840aeff0 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 >> v1:172.21.15.62:6789/0 conn(0x7ff6840aaee0 legacy=0x7ff6840aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 >> v1:172.21.15.19:6789/0 conn(0x7ff6840ae460 legacy=0x7ff6840ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840ac630 con 0x7ff6840aeff0 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:26.536 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff67c001a10 con 0x7ff6840aeff0 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff6837fe640 10 monclient: dump: 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:26.537 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff689b1a640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.553+0000 7ff689b1a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff67c001be0 con 0x7ff6840aeff0 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff6837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff6837fe640 1 -- 172.21.15.19:0/600983479 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff67c001ff0 con 0x7ff6840aeff0 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff6837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff6837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff6837fe640 10 monclient: dump: 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:26.538 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:26.539 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:26.539 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:26.539 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:26.539 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff689b1a640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.554+0000 7ff689b1a640 4 set_mon_vals no callback set 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.555+0000 7ff689b1a640 10 monclient: shutdown 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.556+0000 7ff689b1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.556+0000 7ff689b1a640 1 -- 172.21.15.19:0/600983479 >> v1:172.21.15.19:6790/0 conn(0x7ff6840aeff0 legacy=0x7ff6840af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.556+0000 7ff689b1a640 1 -- 172.21.15.19:0/600983479 shutdown_connections 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.556+0000 7ff689b1a640 1 -- 172.21.15.19:0/600983479 >> 172.21.15.19:0/600983479 conn(0x7ff6840b3a30 msgr2=0x7ff6840b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.556+0000 7ff689b1a640 1 -- 172.21.15.19:0/600983479 shutdown_connections 2024-03-29T11:24:26.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.557+0000 7ff689b1a640 1 -- 172.21.15.19:0/600983479 wait complete. 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.558+0000 7ff689b1a640 10 monclient: build_initial_monmap 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.558+0000 7ff689b1a640 10 monclient: monmap: 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:26.541 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.558+0000 7ff689b1a640 1 librados: starting msgr at 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.558+0000 7ff689b1a640 1 librados: starting objecter 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.559+0000 7ff689b1a640 1 Processor -- start 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.559+0000 7ff689b1a640 1 -- start start 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.559+0000 7ff689b1a640 1 librados: setting wanted keys 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.559+0000 7ff689b1a640 1 librados: calling monclient init 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.559+0000 7ff689b1a640 10 monclient: init 2024-03-29T11:24:26.542 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.560+0000 7ff689b1a640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.560+0000 7ff689b1a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6840aaee0 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 10 monclient(hunting): picked mon.noname-c con 0x7ff6840ae460 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 10 monclient(hunting): picked mon.noname-b con 0x7ff6840aeff0 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6841e1a00 con 0x7ff6840aaee0 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff684070860 con 0x7ff6840aeff0 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6841e3f40 con 0x7ff6840ae460 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:26.543 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.562+0000 7ff689b1a640 10 monclient(hunting): authenticate will time out at 1186.825240s 2024-03-29T11:24:26.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff688b18640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff6840aaee0 0x7ff6840712b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:59925/0 (socket says 172.21.15.19:59925) 2024-03-29T11:24:26.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff688b18640 1 -- 172.21.15.19:0/36230251 learned_addr learned my addr 172.21.15.19:0/36230251 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 10 client.?.objecter ms_handle_connect 0x7ff6840aaee0 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914500074 0 0) 0x7ff6841e1a00 con 0x7ff6840aaee0 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:26.547 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.564+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff668004eb0 con 0x7ff6840aaee0 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2752760380 0 0) 0x7ff668004eb0 con 0x7ff6840aaee0 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6841e1a00 con 0x7ff6840aaee0 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff678001a10 con 0x7ff6840aaee0 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient(hunting): dump: 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:26.548 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:26.549 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:26.549 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:26.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:26.549 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3282927550 0 0) 0x7ff6841e1a00 con 0x7ff6840aaee0 2024-03-29T11:24:26.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 >> v1:172.21.15.19:6790/0 conn(0x7ff6840ae460 legacy=0x7ff6841e4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 >> v1:172.21.15.62:6789/0 conn(0x7ff6840aeff0 legacy=0x7ff6841ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:26.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 monclient: found mon.a 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6841e4b70 con 0x7ff6840aaee0 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff6817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff689b1a640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff689b1a640 10 monclient: _renew_subs 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff689b1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.565+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6841496e0 con 0x7ff6840aaee0 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff678001d10 con 0x7ff6840aaee0 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:26.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff678003a60 con 0x7ff6840aaee0 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6817fa640 10 monclient: dump: 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff6827fc640 4 set_mon_vals no callback set 2024-03-29T11:24:26.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.566+0000 7ff689b1a640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.567+0000 7ff689b1a640 10 monclient: _renew_subs 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.567+0000 7ff689b1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.567+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6841efcf0 con 0x7ff6840aaee0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.567+0000 7ff689b1a640 1 librados: init done 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff678003bc0 con 0x7ff6840aaee0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff689b1a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff689b1a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff689b1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6841e40d0 con 0x7ff6840aaee0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff678054a30 con 0x7ff6840aaee0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff6817fa640 10 client.4416.objecter ms_dispatch 0x7ff684000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff6817fa640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff6817fa640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.568+0000 7ff6817fa640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:24:26.553 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.569+0000 7ff683fff640 10 monclient: get_auth_request con 0x7ff66806db70 auth_method 0 2024-03-29T11:24:26.554 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.569+0000 7ff6817fa640 10 client.4416.objecter ms_handle_connect 0x7ff66806db70 2024-03-29T11:24:26.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.577+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff678050370 con 0x7ff6840aaee0 2024-03-29T11:24:26.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.577+0000 7ff6817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:26.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.577+0000 7ff6817fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:26.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.923+0000 7ff689b1a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:26.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.923+0000 7ff689b1a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:26.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.923+0000 7ff689b1a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:26.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.923+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6841e4d00 con 0x7ff6840aaee0 2024-03-29T11:24:26.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.924+0000 7ff6817fa640 1 -- 172.21.15.19:0/36230251 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff678052470 con 0x7ff6840aaee0 2024-03-29T11:24:26.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.924+0000 7ff6817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:26.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.924+0000 7ff6817fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:26.907 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:26.907 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.928+0000 7ff689b1a640 10 librados: watch_flush enter 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.928+0000 7ff689b1a640 10 librados: watch_flush exit 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.928+0000 7ff689b1a640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.928+0000 7ff689b1a640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.929+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff66806db70 legacy=0x7ff668070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.929+0000 7ff689b1a640 10 monclient: shutdown 2024-03-29T11:24:26.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.929+0000 7ff689b1a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:26.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.929+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 >> v1:172.21.15.19:6789/0 conn(0x7ff6840aaee0 legacy=0x7ff6840712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:26.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.932+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 shutdown_connections 2024-03-29T11:24:26.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.932+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 >> 172.21.15.19:0/36230251 conn(0x7ff6840b3a30 msgr2=0x7ff684069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:26.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.933+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 shutdown_connections 2024-03-29T11:24:26.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.933+0000 7ff689b1a640 1 -- 172.21.15.19:0/36230251 wait complete. 2024-03-29T11:24:26.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:26.933+0000 7ff689b1a640 1 librados: shutdown 2024-03-29T11:24:26.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:33.741 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:33.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.909+0000 7fba9f23e640 10 monclient: get_monmap_and_config 2024-03-29T11:24:33.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.909+0000 7fba9f23e640 10 monclient: build_initial_monmap 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.909+0000 7fba9f23e640 10 monclient: monmap: 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:33.911407+0000 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:33.911407+0000 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:33.893 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:33.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.910+0000 7fba9f23e640 1 Processor -- start 2024-03-29T11:24:33.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.910+0000 7fba9f23e640 1 -- start start 2024-03-29T11:24:33.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.910+0000 7fba9f23e640 10 monclient: init 2024-03-29T11:24:33.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:33.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:33.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient(hunting): picked mon.noname-c con 0x7fba980b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient(hunting): picked mon.noname-b con 0x7fba9814a700 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient(hunting): picked mon.noname-a con 0x7fba980ac5a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980aba70 con 0x7fba980ac5a0 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980aecb0 con 0x7fba980b0770 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba980af150 con 0x7fba9814a700 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:33.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.911+0000 7fba9f23e640 10 monclient(hunting): authenticate will time out at 1194.175757s 2024-03-29T11:24:33.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.914+0000 7fba9cfda640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fba980b0770 0x7fba98149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:60161/0 (socket says 172.21.15.19:60161) 2024-03-29T11:24:33.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.914+0000 7fba9cfda640 1 -- 172.21.15.19:0/2236598767 learned_addr learned my addr 172.21.15.19:0/2236598767 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.915+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541855340 0 0) 0x7fba980aba70 con 0x7fba980ac5a0 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.915+0000 7fba977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.915+0000 7fba977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.915+0000 7fba977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.915+0000 7fba977fe640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.915+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba80003520 con 0x7fba980ac5a0 2024-03-29T11:24:33.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94370875 0 0) 0x7fba80003520 con 0x7fba980ac5a0 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 >> v1:172.21.15.19:6790/0 conn(0x7fba9814a700 legacy=0x7fba980abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 >> v1:172.21.15.62:6789/0 conn(0x7fba980b0770 legacy=0x7fba98149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba980ac9d0 con 0x7fba980ac5a0 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fba88001670 con 0x7fba980ac5a0 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:33.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: dump: 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba880018d0 con 0x7fba980ac5a0 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 1 -- 172.21.15.19:0/2236598767 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fba88001670 con 0x7fba980ac5a0 2024-03-29T11:24:33.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.916+0000 7fba977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.917+0000 7fba977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.917+0000 7fba977fe640 10 monclient: dump: 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:33.903 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:33.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.917+0000 7fba9f23e640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.917+0000 7fba9f23e640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.917+0000 7fba9f23e640 4 set_mon_vals no callback set 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.918+0000 7fba9f23e640 10 monclient: shutdown 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.918+0000 7fba9f23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.919+0000 7fba9f23e640 1 -- 172.21.15.19:0/2236598767 >> v1:172.21.15.19:6789/0 conn(0x7fba980ac5a0 legacy=0x7fba98151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.919+0000 7fba9f23e640 1 -- 172.21.15.19:0/2236598767 shutdown_connections 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.919+0000 7fba9f23e640 1 -- 172.21.15.19:0/2236598767 >> 172.21.15.19:0/2236598767 conn(0x7fba9806b660 msgr2=0x7fba9806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:33.905 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.919+0000 7fba9f23e640 1 -- 172.21.15.19:0/2236598767 shutdown_connections 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.923+0000 7fba9f23e640 1 -- 172.21.15.19:0/2236598767 wait complete. 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.923+0000 7fba9f23e640 10 monclient: build_initial_monmap 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 10 monclient: monmap: 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:33.906 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:33.907 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:33.907 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:33.907 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:33.907 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 1 librados: starting msgr at 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 1 librados: starting objecter 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 1 Processor -- start 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 1 -- start start 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 1 librados: setting wanted keys 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 1 librados: calling monclient init 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.924+0000 7fba9f23e640 10 monclient: init 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.925+0000 7fba9f23e640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.925+0000 7fba9f23e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:33.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.925+0000 7fba9f23e640 10 monclient(hunting): picked mon.noname-c con 0x7fba980ac5a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:33.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba9f23e640 10 monclient(hunting): picked mon.noname-a con 0x7fba980b0770 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:33.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba97fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fba980b0770 0x7fba981dbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:33959/0 (socket says 172.21.15.19:33959) 2024-03-29T11:24:33.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba97fff640 1 -- 172.21.15.19:0/1344258657 learned_addr learned my addr 172.21.15.19:0/1344258657 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:33.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba9f23e640 10 monclient(hunting): picked mon.noname-b con 0x7fba9814a700 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba9f23e640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba98128f00 con 0x7fba980b0770 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba9f23e640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba98128b20 con 0x7fba9814a700 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.929+0000 7fba9f23e640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba98128cb0 con 0x7fba980ac5a0 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.930+0000 7fba9f23e640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.930+0000 7fba9f23e640 10 monclient(hunting): authenticate will time out at 1194.194032s 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.930+0000 7fba957fa640 10 client.?.objecter ms_handle_connect 0x7fba980b0770 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.930+0000 7fba957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:33.913 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.930+0000 7fba957fa640 10 client.?.objecter ms_handle_connect 0x7fba980ac5a0 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.930+0000 7fba957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196968580 0 0) 0x7fba98128cb0 con 0x7fba980ac5a0 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:24:33.914 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba84004eb0 con 0x7fba980ac5a0 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 274643497 0 0) 0x7fba84004eb0 con 0x7fba980ac5a0 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba98128cb0 con 0x7fba980ac5a0 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fba90001890 con 0x7fba980ac5a0 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): dump: 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:33.916 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.931+0000 7fba957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3722017512 0 0) 0x7fba98128cb0 con 0x7fba980ac5a0 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 >> v1:172.21.15.62:6789/0 conn(0x7fba9814a700 legacy=0x7fba981e5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 >> v1:172.21.15.19:6789/0 conn(0x7fba980b0770 legacy=0x7fba981dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:33.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 1 monclient: found mon.c 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba981478f0 con 0x7fba980ac5a0 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 10 monclient: _renew_subs 2024-03-29T11:24:33.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:33.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba98126fc0 con 0x7fba980ac5a0 2024-03-29T11:24:33.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:33.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 10 monclient: _renew_subs 2024-03-29T11:24:33.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:33.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba98127f30 con 0x7fba980ac5a0 2024-03-29T11:24:33.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.932+0000 7fba9f23e640 1 librados: init done 2024-03-29T11:24:33.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.934+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba90001a60 con 0x7fba980ac5a0 2024-03-29T11:24:33.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.934+0000 7fba957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:33.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.934+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fba90004e90 con 0x7fba980ac5a0 2024-03-29T11:24:33.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.934+0000 7fba957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:33.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.934+0000 7fba957fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.934+0000 7fba957fa640 10 monclient: dump: 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.935+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fba9001ae60 con 0x7fba980ac5a0 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fba90084de0 con 0x7fba980ac5a0 2024-03-29T11:24:33.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba957fa640 10 client.4430.objecter ms_dispatch 0x7fba98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba957fa640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba7effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba7effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba7effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba980ab560 con 0x7fba980ac5a0 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba957fa640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba957fa640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba96ffd640 4 set_mon_vals no callback set 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.936+0000 7fba97fff640 10 monclient: get_auth_request con 0x7fba8406dd00 auth_method 0 2024-03-29T11:24:33.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.937+0000 7fba957fa640 10 client.4430.objecter ms_handle_connect 0x7fba8406dd00 2024-03-29T11:24:33.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.944+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba900506c0 con 0x7fba980ac5a0 2024-03-29T11:24:33.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.944+0000 7fba957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:33.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:33.944+0000 7fba957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:34.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.280+0000 7fba7effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:34.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.280+0000 7fba7effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:34.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.280+0000 7fba7effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:24:34.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.280+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba98075f10 con 0x7fba980ac5a0 2024-03-29T11:24:34.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.283+0000 7fba957fa640 1 -- 172.21.15.19:0/1344258657 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fba900525f0 con 0x7fba980ac5a0 2024-03-29T11:24:34.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.283+0000 7fba957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:34.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.283+0000 7fba957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:34.266 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:34.266 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.286+0000 7fba7effd640 10 librados: watch_flush enter 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.286+0000 7fba7effd640 10 librados: watch_flush exit 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.286+0000 7fba7effd640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.286+0000 7fba7effd640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.286+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 >> v1:172.21.15.19:6816/3721232097 conn(0x7fba8406dd00 legacy=0x7fba840701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 10 monclient: shutdown 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 >> v1:172.21.15.19:6790/0 conn(0x7fba980ac5a0 legacy=0x7fba981db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:34.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 shutdown_connections 2024-03-29T11:24:34.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 >> 172.21.15.19:0/1344258657 conn(0x7fba9806b660 msgr2=0x7fba980abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:34.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 shutdown_connections 2024-03-29T11:24:34.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 1 -- 172.21.15.19:0/1344258657 wait complete. 2024-03-29T11:24:34.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:34.287+0000 7fba7effd640 1 librados: shutdown 2024-03-29T11:24:34.287 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:41.094 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:41.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.263+0000 7f618e6c3640 10 monclient: get_monmap_and_config 2024-03-29T11:24:41.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.263+0000 7f618e6c3640 10 monclient: build_initial_monmap 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.263+0000 7f618e6c3640 10 monclient: monmap: 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:41.265254+0000 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:41.265254+0000 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:41.247 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:41.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.265+0000 7f618e6c3640 1 Processor -- start 2024-03-29T11:24:41.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.265+0000 7f618e6c3640 1 -- start start 2024-03-29T11:24:41.250 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.265+0000 7f618e6c3640 10 monclient: init 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient(hunting): picked mon.noname-a con 0x7f6180006e30 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient(hunting): picked mon.noname-c con 0x7f6180035e00 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient(hunting): picked mon.noname-b con 0x7f61800c9670 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180035970 con 0x7f6180006e30 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180036280 con 0x7f6180035e00 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180035640 con 0x7f61800c9670 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:41.251 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.266+0000 7f618e6c3640 10 monclient(hunting): authenticate will time out at 1201.530417s 2024-03-29T11:24:41.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f618cec0640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f6180035e00 0x7f61800c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:51023/0 (socket says 172.21.15.19:51023) 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f618cec0640 1 -- 172.21.15.19:0/1384492565 learned_addr learned my addr 172.21.15.19:0/1384492565 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780440757 0 0) 0x7f6180036280 con 0x7f6180035e00 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f617ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f617ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f617ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f617ffff640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.271+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6174003520 con 0x7f6180035e00 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095912961 0 0) 0x7f6174003520 con 0x7f6180035e00 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 >> v1:172.21.15.19:6790/0 conn(0x7f61800c9670 legacy=0x7f61800d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 >> v1:172.21.15.19:6789/0 conn(0x7f6180006e30 legacy=0x7f61800cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:41.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 monclient: found mon.noname-c 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180034e20 con 0x7f6180035e00 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f618e6c3640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f618e6c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f618804f730 con 0x7f6180035e00 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: dump: 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:41.259 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f618e6c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6188075f10 con 0x7f6180035e00 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 1 -- 172.21.15.19:0/1384492565 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f61880758b0 con 0x7f6180035e00 2024-03-29T11:24:41.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f617ffff640 10 monclient: dump: 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f618e6c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:41.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.272+0000 7f618e6c3640 4 set_mon_vals no callback set 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.273+0000 7f618e6c3640 10 monclient: shutdown 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.273+0000 7f618e6c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.273+0000 7f618e6c3640 1 -- 172.21.15.19:0/1384492565 >> v1:172.21.15.62:6789/0 conn(0x7f6180035e00 legacy=0x7f61800c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.273+0000 7f618e6c3640 1 -- 172.21.15.19:0/1384492565 shutdown_connections 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.273+0000 7f618e6c3640 1 -- 172.21.15.19:0/1384492565 >> 172.21.15.19:0/1384492565 conn(0x7f6180005c00 msgr2=0x7f6180006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.274+0000 7f618e6c3640 1 -- 172.21.15.19:0/1384492565 shutdown_connections 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.274+0000 7f618e6c3640 1 -- 172.21.15.19:0/1384492565 wait complete. 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.274+0000 7f618e6c3640 10 monclient: build_initial_monmap 2024-03-29T11:24:41.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.274+0000 7f618e6c3640 10 monclient: monmap: 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 1 librados: starting msgr at 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 1 librados: starting objecter 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 1 Processor -- start 2024-03-29T11:24:41.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 1 -- start start 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 1 librados: setting wanted keys 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 1 librados: calling monclient init 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.275+0000 7f618e6c3640 10 monclient: init 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient(hunting): picked mon.noname-b con 0x7f6180006e30 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient(hunting): picked mon.noname-c con 0x7f6180035e00 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient(hunting): picked mon.noname-a con 0x7f61800c9670 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180033b90 con 0x7f61800c9670 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180182d20 con 0x7f6180006e30 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6180183000 con 0x7f6180035e00 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:41.264 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.276+0000 7f618e6c3640 10 monclient(hunting): authenticate will time out at 1201.540302s 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f618dec2640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f61800c9670 0x7f618002f8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:59391/0 (socket says 172.21.15.19:59391) 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f618dec2640 1 -- 172.21.15.19:0/2452946758 learned_addr learned my addr 172.21.15.19:0/2452946758 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 client.?.objecter ms_handle_connect 0x7f61800c9670 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238323956 0 0) 0x7f6180033b90 con 0x7f61800c9670 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:41.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f616c004eb0 con 0x7f61800c9670 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 12131196 0 0) 0x7f616c004eb0 con 0x7f61800c9670 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6180033b90 con 0x7f61800c9670 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6184001890 con 0x7f61800c9670 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): dump: 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:41.266 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.281+0000 7f617dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.282+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4068046797 0 0) 0x7f6180033b90 con 0x7f61800c9670 2024-03-29T11:24:41.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.282+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 >> v1:172.21.15.19:6790/0 conn(0x7f6180035e00 legacy=0x7f6180183490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:41.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 >> v1:172.21.15.62:6789/0 conn(0x7f6180006e30 legacy=0x7f61800327c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:41.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:41.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 1 monclient: found mon.a 2024-03-29T11:24:41.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180033de0 con 0x7f61800c9670 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f617dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 10 monclient: _renew_subs 2024-03-29T11:24:41.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:41.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6180183ba0 con 0x7f61800c9670 2024-03-29T11:24:41.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:41.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 10 monclient: _renew_subs 2024-03-29T11:24:41.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:41.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61800301f0 con 0x7f61800c9670 2024-03-29T11:24:41.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.285+0000 7f618e6c3640 1 librados: init done 2024-03-29T11:24:41.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.289+0000 7f618e6c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:41.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.289+0000 7f618e6c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:41.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.289+0000 7f618e6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:41.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.289+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6180007b90 con 0x7f61800c9670 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61840019f0 con 0x7f61800c9670 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f61840037f0 con 0x7f61800c9670 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 10 monclient: dump: 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:41.276 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f6184003a10 con 0x7f61800c9670 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f6184084b30 con 0x7f61800c9670 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 10 client.4422.objecter ms_dispatch 0x7f6188000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617dffb640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.290+0000 7f617f7fe640 4 set_mon_vals no callback set 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.292+0000 7f618d6c1640 10 monclient: get_auth_request con 0x7f616c06dcd0 auth_method 0 2024-03-29T11:24:41.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.292+0000 7f617dffb640 10 client.4422.objecter ms_handle_connect 0x7f616c06dcd0 2024-03-29T11:24:41.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.300+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6184050490 con 0x7f61800c9670 2024-03-29T11:24:41.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.300+0000 7f617dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:41.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.300+0000 7f617dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:41.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.635+0000 7f618e6c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:41.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.635+0000 7f618e6c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:41.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.635+0000 7f618e6c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:41.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.635+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61801831b0 con 0x7f61800c9670 2024-03-29T11:24:41.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.636+0000 7f617dffb640 1 -- 172.21.15.19:0/2452946758 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f61801831b0 con 0x7f61800c9670 2024-03-29T11:24:41.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.636+0000 7f617dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:41.619 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.636+0000 7f617dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:41.622 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:41.622 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.642+0000 7f618e6c3640 10 librados: watch_flush enter 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.642+0000 7f618e6c3640 10 librados: watch_flush exit 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.642+0000 7f618e6c3640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.642+0000 7f618e6c3640 10 client.4422.objecter successfully canceled tick 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.645+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 >> v1:172.21.15.19:6816/3721232097 conn(0x7f616c06dcd0 legacy=0x7f616c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.645+0000 7f618e6c3640 10 monclient: shutdown 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.645+0000 7f618e6c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:41.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.645+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 >> v1:172.21.15.19:6789/0 conn(0x7f61800c9670 legacy=0x7f618002f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:41.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.645+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 shutdown_connections 2024-03-29T11:24:41.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.645+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 >> 172.21.15.19:0/2452946758 conn(0x7f6180005c00 msgr2=0x7f61800c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:41.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.646+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 shutdown_connections 2024-03-29T11:24:41.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.646+0000 7f618e6c3640 1 -- 172.21.15.19:0/2452946758 wait complete. 2024-03-29T11:24:41.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:41.646+0000 7f618e6c3640 1 librados: shutdown 2024-03-29T11:24:41.646 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:48.453 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:48.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.619+0000 7f51e8d05640 10 monclient: get_monmap_and_config 2024-03-29T11:24:48.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.620+0000 7f51e8d05640 10 monclient: build_initial_monmap 2024-03-29T11:24:48.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.620+0000 7f51e8d05640 10 monclient: monmap: 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:48.621821+0000 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:48.621821+0000 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:48.604 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.623+0000 7f51e8d05640 1 Processor -- start 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.623+0000 7f51e8d05640 1 -- start start 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.623+0000 7f51e8d05640 10 monclient: init 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient(hunting): picked mon.noname-a con 0x7f51e40b2800 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient(hunting): picked mon.noname-c con 0x7f51e40b3220 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient(hunting): picked mon.noname-b con 0x7f51e41439c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40b3840 con 0x7f51e40b2800 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e406f230 con 0x7f51e40b3220 2024-03-29T11:24:48.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e406ef50 con 0x7f51e41439c0 2024-03-29T11:24:48.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:48.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.624+0000 7f51e8d05640 10 monclient(hunting): authenticate will time out at 1208.888935s 2024-03-29T11:24:48.611 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.628+0000 7f51e3fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f51e41439c0 0x7f51e4151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:34031/0 (socket says 172.21.15.19:34031) 2024-03-29T11:24:48.611 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.628+0000 7f51e3fff640 1 -- 172.21.15.19:0/159649854 learned_addr learned my addr 172.21.15.19:0/159649854 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:48.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026202047 0 0) 0x7f51e406ef50 con 0x7f51e41439c0 2024-03-29T11:24:48.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:48.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:48.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:48.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:24:48.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51cc003520 con 0x7f51e41439c0 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2222257100 0 0) 0x7f51cc003520 con 0x7f51e41439c0 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 >> v1:172.21.15.62:6789/0 conn(0x7f51e40b3220 legacy=0x7f51e4146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 >> v1:172.21.15.19:6789/0 conn(0x7f51e40b2800 legacy=0x7f51e41432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 monclient: found mon.noname-b 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e4071d30 con 0x7f51e41439c0 2024-03-29T11:24:48.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f51dc001a10 con 0x7f51e41439c0 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.629+0000 7f51e27fc640 10 monclient: dump: 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:48.614 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:48.615 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:48.615 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:48.615 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:48.615 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:48.616 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.632+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51dc001b70 con 0x7f51e41439c0 2024-03-29T11:24:48.616 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.632+0000 7f51e27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:48.616 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.632+0000 7f51e27fc640 1 -- 172.21.15.19:0/159649854 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f51dc001f80 con 0x7f51e41439c0 2024-03-29T11:24:48.616 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.632+0000 7f51e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.632+0000 7f51e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.632+0000 7f51e27fc640 10 monclient: dump: 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.633+0000 7f51e8d05640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:24:48.617 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.633+0000 7f51e8d05640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:48.618 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.633+0000 7f51e8d05640 4 set_mon_vals no callback set 2024-03-29T11:24:48.621 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.637+0000 7f51e8d05640 10 monclient: shutdown 2024-03-29T11:24:48.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.637+0000 7f51e8d05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:48.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.637+0000 7f51e8d05640 1 -- 172.21.15.19:0/159649854 >> v1:172.21.15.19:6790/0 conn(0x7f51e41439c0 legacy=0x7f51e4151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.638+0000 7f51e8d05640 1 -- 172.21.15.19:0/159649854 shutdown_connections 2024-03-29T11:24:48.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.638+0000 7f51e8d05640 1 -- 172.21.15.19:0/159649854 >> 172.21.15.19:0/159649854 conn(0x7f51e4066a00 msgr2=0x7f51e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:48.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.639+0000 7f51e8d05640 1 -- 172.21.15.19:0/159649854 shutdown_connections 2024-03-29T11:24:48.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.641+0000 7f51e8d05640 1 -- 172.21.15.19:0/159649854 wait complete. 2024-03-29T11:24:48.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.642+0000 7f51e8d05640 10 monclient: build_initial_monmap 2024-03-29T11:24:48.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.642+0000 7f51e8d05640 10 monclient: monmap: 2024-03-29T11:24:48.626 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:48.626 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.642+0000 7f51e8d05640 1 librados: starting msgr at 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.642+0000 7f51e8d05640 1 librados: starting objecter 2024-03-29T11:24:48.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.642+0000 7f51e8d05640 1 Processor -- start 2024-03-29T11:24:48.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.643+0000 7f51e8d05640 1 -- start start 2024-03-29T11:24:48.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.643+0000 7f51e8d05640 1 librados: setting wanted keys 2024-03-29T11:24:48.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.643+0000 7f51e8d05640 1 librados: calling monclient init 2024-03-29T11:24:48.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.643+0000 7f51e8d05640 10 monclient: init 2024-03-29T11:24:48.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.644+0000 7f51e8d05640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:48.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.644+0000 7f51e8d05640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:48.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.647+0000 7f51e8d05640 10 monclient(hunting): picked mon.noname-b con 0x7f51e40b2800 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:48.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.647+0000 7f51e37fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f51e40b2800 0x7f51e413eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:57139/0 (socket says 172.21.15.19:57139) 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 10 monclient(hunting): picked mon.noname-a con 0x7f51e40b3220 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 10 monclient(hunting): picked mon.noname-c con 0x7f51e41439c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40ab050 con 0x7f51e40b3220 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40a94e0 con 0x7f51e40b2800 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e40a97c0 con 0x7f51e41439c0 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e8d05640 10 monclient(hunting): authenticate will time out at 1208.912515s 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51e37fe640 1 -- 172.21.15.19:0/1748066282 learned_addr learned my addr 172.21.15.19:0/1748066282 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51cbfff640 10 client.?.objecter ms_handle_connect 0x7f51e40b2800 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.648+0000 7f51cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249060367 0 0) 0x7f51e40a94e0 con 0x7f51e40b2800 2024-03-29T11:24:48.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d0004eb0 con 0x7f51e40b2800 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1738299170 0 0) 0x7f51d0004eb0 con 0x7f51e40b2800 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51e40a94e0 con 0x7f51e40b2800 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f51d8001940 con 0x7f51e40b2800 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): dump: 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.634 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.649+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3249861254 0 0) 0x7f51e40a94e0 con 0x7f51e40b2800 2024-03-29T11:24:48.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.650+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 >> v1:172.21.15.19:6790/0 conn(0x7f51e41439c0 legacy=0x7f51e41e7980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:48.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 >> v1:172.21.15.19:6789/0 conn(0x7f51e40b3220 legacy=0x7f51e41e7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:48.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 1 monclient: found mon.b 2024-03-29T11:24:48.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:48.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:48.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e413f5f0 con 0x7f51e40b2800 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 client.?.objecter ms_handle_connect 0x7f51e40b3220 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 10 monclient: _renew_subs 2024-03-29T11:24:48.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e40a9f80 con 0x7f51e40b2800 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 10 monclient: _renew_subs 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e40b12c0 con 0x7f51e40b2800 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e8d05640 1 librados: init done 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51d8001c60 con 0x7f51e40b2800 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f51d8003a60 con 0x7f51e40b2800 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51cbfff640 10 monclient: dump: 2024-03-29T11:24:48.642 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.652+0000 7f51e1ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.654+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f51d8003bc0 con 0x7f51e40b2800 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.655+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f51d8084b20 con 0x7f51e40b2800 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.655+0000 7f51cbfff640 10 client.4411.objecter ms_dispatch 0x7f51e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:48.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.655+0000 7f51cbfff640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.655+0000 7f51cbfff640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.655+0000 7f51cbfff640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.655+0000 7f51e2ffd640 10 monclient: get_auth_request con 0x7f51d006db80 auth_method 0 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.656+0000 7f51cbfff640 10 client.4411.objecter ms_handle_connect 0x7f51d006db80 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.659+0000 7f51e8d05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.659+0000 7f51e8d05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.659+0000 7f51e8d05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:48.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.659+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c4005080 con 0x7f51e40b2800 2024-03-29T11:24:48.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.670+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f51d8050400 con 0x7f51e40b2800 2024-03-29T11:24:48.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.670+0000 7f51cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:48.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.670+0000 7f51cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:48.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.998+0000 7f51e8d05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:48.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.999+0000 7f51e8d05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:48.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.999+0000 7f51e8d05640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:24:48.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:48.999+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51c4005330 con 0x7f51e40b2800 2024-03-29T11:24:48.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.000+0000 7f51cbfff640 1 -- 172.21.15.19:0/1748066282 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f51d8052330 con 0x7f51e40b2800 2024-03-29T11:24:48.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.000+0000 7f51cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:48.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.000+0000 7f51cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:48.986 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:48.986 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.003+0000 7f51e8d05640 10 librados: watch_flush enter 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.003+0000 7f51e8d05640 10 librados: watch_flush exit 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.003+0000 7f51e8d05640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.003+0000 7f51e8d05640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.004+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 >> v1:172.21.15.19:6816/3721232097 conn(0x7f51d006db80 legacy=0x7f51d0070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.004+0000 7f51e8d05640 10 monclient: shutdown 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.004+0000 7f51e8d05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:48.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.004+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 >> v1:172.21.15.62:6789/0 conn(0x7f51e40b2800 legacy=0x7f51e413eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:48.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.005+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 shutdown_connections 2024-03-29T11:24:48.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.005+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 >> 172.21.15.19:0/1748066282 conn(0x7f51e4066a00 msgr2=0x7f51e4141de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:48.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.006+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 shutdown_connections 2024-03-29T11:24:48.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.007+0000 7f51e8d05640 1 -- 172.21.15.19:0/1748066282 wait complete. 2024-03-29T11:24:48.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:49.007+0000 7f51e8d05640 1 librados: shutdown 2024-03-29T11:24:49.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:55.816 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:55.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.981+0000 7f509d0fb640 10 monclient: get_monmap_and_config 2024-03-29T11:24:55.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.981+0000 7f509d0fb640 10 monclient: build_initial_monmap 2024-03-29T11:24:55.964 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.981+0000 7f509d0fb640 10 monclient: monmap: 2024-03-29T11:24:55.964 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:24:55.982402+0000 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:24:55.982402+0000 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:24:55.965 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.982+0000 7f509d0fb640 1 Processor -- start 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.982+0000 7f509d0fb640 1 -- start start 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.982+0000 7f509d0fb640 10 monclient: init 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.983+0000 7f509d0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.983+0000 7f509d0fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.983+0000 7f509d0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f50981457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.983+0000 7f509d0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f509813f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.983+0000 7f509d0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f50980a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509d0fb640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50980aaef0 con 0x7f50981457f0 2024-03-29T11:24:55.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509d0fb640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5098145bd0 con 0x7f509813f650 2024-03-29T11:24:55.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509d0fb640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5098145eb0 con 0x7f50980a9d70 2024-03-29T11:24:55.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509d0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:55.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509d0fb640 10 monclient(hunting): authenticate will time out at 1216.247042s 2024-03-29T11:24:55.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509759e640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f50980a9d70 0x7f509814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:38142/0 (socket says 172.21.15.19:38142) 2024-03-29T11:24:55.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f509759e640 1 -- 172.21.15.19:0/1579403544 learned_addr learned my addr 172.21.15.19:0/1579403544 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038478371 0 0) 0x7f5098145eb0 con 0x7f50980a9d70 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5080003520 con 0x7f50980a9d70 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398896026 0 0) 0x7f5098145bd0 con 0x7f509813f650 2024-03-29T11:24:55.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.984+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098145eb0 con 0x7f509813f650 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.985+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323584308 0 0) 0x7f50980aaef0 con 0x7f50981457f0 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.985+0000 7f5095d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.985+0000 7f5095d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.985+0000 7f5095d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.985+0000 7f5095d9b640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5098145bd0 con 0x7f50981457f0 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3100788744 0 0) 0x7f5098145eb0 con 0x7f509813f650 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 >> v1:172.21.15.19:6790/0 conn(0x7f50980a9d70 legacy=0x7f509814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 >> v1:172.21.15.19:6789/0 conn(0x7f50981457f0 legacy=0x7f509813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 monclient: found mon.noname-c 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098146340 con 0x7f509813f650 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f508c0015a0 con 0x7f509813f650 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f5095d9b640 10 monclient: dump: 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:55.973 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f509d0fb640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.986+0000 7f509d0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f508c001700 con 0x7f509813f650 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f5095d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f5095d9b640 1 -- 172.21.15.19:0/1579403544 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f508c001b50 con 0x7f509813f650 2024-03-29T11:24:55.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f509d0fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f509d0fb640 4 set_mon_vals no callback set 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f5095d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f5095d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f5095d9b640 10 monclient: dump: 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:55.975 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f509d0fb640 10 monclient: shutdown 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f509d0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.987+0000 7f509d0fb640 1 -- 172.21.15.19:0/1579403544 >> v1:172.21.15.62:6789/0 conn(0x7f509813f650 legacy=0x7f50980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.988+0000 7f509d0fb640 1 -- 172.21.15.19:0/1579403544 shutdown_connections 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.988+0000 7f509d0fb640 1 -- 172.21.15.19:0/1579403544 >> 172.21.15.19:0/1579403544 conn(0x7f5098066a00 msgr2=0x7f5098066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.988+0000 7f509d0fb640 1 -- 172.21.15.19:0/1579403544 shutdown_connections 2024-03-29T11:24:55.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.988+0000 7f509d0fb640 1 -- 172.21.15.19:0/1579403544 wait complete. 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.989+0000 7f509d0fb640 10 monclient: build_initial_monmap 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.989+0000 7f509d0fb640 10 monclient: monmap: 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:55.984 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.990+0000 7f509d0fb640 1 librados: starting msgr at 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.990+0000 7f509d0fb640 1 librados: starting objecter 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.990+0000 7f509d0fb640 1 Processor -- start 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.991+0000 7f509d0fb640 1 -- start start 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.991+0000 7f509d0fb640 1 librados: setting wanted keys 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.991+0000 7f509d0fb640 1 librados: calling monclient init 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.991+0000 7f509d0fb640 10 monclient: init 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.993+0000 7f509d0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.993+0000 7f509d0fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:55.985 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.993+0000 7f509d0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f50980a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.993+0000 7f509d0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f509813f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.993+0000 7f509d0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f50981457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.993+0000 7f509659c640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f509813f650 0x7f5098138ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:36798/0 (socket says 172.21.15.19:36798) 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f509659c640 1 -- 172.21.15.19:0/2590864730 learned_addr learned my addr 172.21.15.19:0/2590864730 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f507f7fe640 10 client.?.objecter ms_handle_connect 0x7f509813f650 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f507f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f507f7fe640 10 client.?.objecter ms_handle_connect 0x7f50980a9d70 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f507f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f507f7fe640 10 client.?.objecter ms_handle_connect 0x7f50981457f0 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f507f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50981ce240 con 0x7f509813f650 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.994+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50981398d0 con 0x7f50981457f0 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50981cf080 con 0x7f50980a9d70 2024-03-29T11:24:55.986 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f509d0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100007065 0 0) 0x7f50981ce240 con 0x7f509813f650 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f509d0fb640 10 monclient(hunting): authenticate will time out at 1216.258302s 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f507f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f507f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f507f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f507f7fe640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.995+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5084004eb0 con 0x7f509813f650 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425849009 0 0) 0x7f50981398d0 con 0x7f50981457f0 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50981ce240 con 0x7f50981457f0 2024-03-29T11:24:55.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245599498 0 0) 0x7f50981cf080 con 0x7f50980a9d70 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.996+0000 7f507f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50981398d0 con 0x7f50980a9d70 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096568413 0 0) 0x7f5084004eb0 con 0x7f509813f650 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50981cf080 con 0x7f509813f650 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3849885669 0 0) 0x7f50981ce240 con 0x7f50981457f0 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5084004eb0 con 0x7f50981457f0 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f508c0015a0 con 0x7f509813f650 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.997+0000 7f507f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:55.988 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f50900014d0 con 0x7f50981457f0 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:55.989 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3068303605 0 0) 0x7f50981cf080 con 0x7f509813f650 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 >> v1:172.21.15.19:6790/0 conn(0x7f50980a9d70 legacy=0x7f50981ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 >> v1:172.21.15.62:6789/0 conn(0x7f50981457f0 legacy=0x7f50981391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 1 monclient: found mon.a 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.998+0000 7f507f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f507f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50981cd8d0 con 0x7f509813f650 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f507f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f507f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f507f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 10 monclient: _renew_subs 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50981cdac0 con 0x7f509813f650 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:55.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 10 monclient: _renew_subs 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50981cd5f0 con 0x7f509813f650 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:55.999+0000 7f509d0fb640 1 librados: init done 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f508c0039c0 con 0x7f509813f650 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f507f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f508c003d50 con 0x7f509813f650 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f507f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f507f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f509559a640 4 set_mon_vals no callback set 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.000+0000 7f507f7fe640 10 monclient: dump: 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:24:55.992 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:55.993 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:24:55.993 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:24:55.993 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:24:55.993 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:24:55.993 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:24:55.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.002+0000 7f509d0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:55.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.002+0000 7f509d0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:55.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.002+0000 7f509d0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.002+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5078005080 con 0x7f509813f650 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.003+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f508c003eb0 con 0x7f509813f650 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.004+0000 7f5096d9d640 10 monclient: get_auth_request con 0x7f508406de40 auth_method 0 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.005+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f508c084b60 con 0x7f509813f650 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.005+0000 7f507f7fe640 10 client.4434.objecter ms_dispatch 0x7f5098000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.005+0000 7f507f7fe640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.005+0000 7f507f7fe640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.005+0000 7f507f7fe640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:24:55.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.005+0000 7f507f7fe640 10 client.4434.objecter ms_handle_connect 0x7f508406de40 2024-03-29T11:24:55.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.014+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f508c050440 con 0x7f509813f650 2024-03-29T11:24:55.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.014+0000 7f507f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:55.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.014+0000 7f507f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:56.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.415+0000 7f509d0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:56.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.415+0000 7f509d0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:56.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.415+0000 7f509d0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:24:56.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.415+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5078005330 con 0x7f509813f650 2024-03-29T11:24:56.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.418+0000 7f507f7fe640 1 -- 172.21.15.19:0/2590864730 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f508c052370 con 0x7f509813f650 2024-03-29T11:24:56.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.418+0000 7f507f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:56.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.418+0000 7f507f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:56.401 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:24:56.401 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.420+0000 7f509d0fb640 10 librados: watch_flush enter 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.420+0000 7f509d0fb640 10 librados: watch_flush exit 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.420+0000 7f509d0fb640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.420+0000 7f509d0fb640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 >> v1:172.21.15.19:6816/3721232097 conn(0x7f508406de40 legacy=0x7f5084070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 10 monclient: shutdown 2024-03-29T11:24:56.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:56.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 >> v1:172.21.15.19:6789/0 conn(0x7f509813f650 legacy=0x7f5098138ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:56.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 shutdown_connections 2024-03-29T11:24:56.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 >> 172.21.15.19:0/2590864730 conn(0x7f5098066a00 msgr2=0x7f5098066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:56.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 shutdown_connections 2024-03-29T11:24:56.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 -- 172.21.15.19:0/2590864730 wait complete. 2024-03-29T11:24:56.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:24:56.421+0000 7f509d0fb640 1 librados: shutdown 2024-03-29T11:24:56.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:03.226 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:03.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.380+0000 7f0907fff640 10 monclient: get_monmap_and_config 2024-03-29T11:25:03.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.380+0000 7f0907fff640 10 monclient: build_initial_monmap 2024-03-29T11:25:03.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.380+0000 7f0907fff640 10 monclient: monmap: 2024-03-29T11:25:03.363 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:03.363 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:03.363 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:03.380887+0000 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:03.380887+0000 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:03.364 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:03.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.383+0000 7f0907fff640 1 Processor -- start 2024-03-29T11:25:03.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.383+0000 7f0907fff640 1 -- start start 2024-03-29T11:25:03.366 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.383+0000 7f0907fff640 10 monclient: init 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient(hunting): picked mon.noname-a con 0x7f09080a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient(hunting): picked mon.noname-c con 0x7f09080add40 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient(hunting): picked mon.noname-b con 0x7f09080aad20 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0908149870 con 0x7f09080a9750 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09080abf80 con 0x7f09080add40 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0908149ac0 con 0x7f09080aad20 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:03.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0907fff640 10 monclient(hunting): authenticate will time out at 1223.648349s 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f09077fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f09080aad20 0x7f0908151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:56708/0 (socket says 172.21.15.19:56708) 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f09077fe640 1 -- 172.21.15.19:0/1672366861 learned_addr learned my addr 172.21.15.19:0/1672366861 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74797204 0 0) 0x7f0908149ac0 con 0x7f09080aad20 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0905ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0905ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0905ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0905ffb640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.385+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f0003520 con 0x7f09080aad20 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.386+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 565483214 0 0) 0x7f08f0003520 con 0x7f09080aad20 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.386+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 >> v1:172.21.15.62:6789/0 conn(0x7f09080add40 legacy=0x7f0908143610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:03.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.386+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 >> v1:172.21.15.19:6789/0 conn(0x7f09080a9750 legacy=0x7f0908149fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:03.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:03.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 1 monclient: found mon.noname-b 2024-03-29T11:25:03.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:03.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:25:03.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090814a6f0 con 0x7f09080aad20 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0907fff640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0907fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f08f8001690 con 0x7f09080aad20 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: dump: 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.374 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0907fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08f80019a0 con 0x7f09080aad20 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0905ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0907fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.388+0000 7f0907fff640 4 set_mon_vals no callback set 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0905ffb640 1 -- 172.21.15.19:0/1672366861 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f08f8001d30 con 0x7f09080aad20 2024-03-29T11:25:03.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0905ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0905ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0905ffb640 10 monclient: dump: 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:03.376 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 10 monclient: shutdown 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 1 -- 172.21.15.19:0/1672366861 >> v1:172.21.15.19:6790/0 conn(0x7f09080aad20 legacy=0x7f0908151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 1 -- 172.21.15.19:0/1672366861 shutdown_connections 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 1 -- 172.21.15.19:0/1672366861 >> 172.21.15.19:0/1672366861 conn(0x7f09080673e0 msgr2=0x7f09080b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 1 -- 172.21.15.19:0/1672366861 shutdown_connections 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.389+0000 7f0907fff640 1 -- 172.21.15.19:0/1672366861 wait complete. 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 10 monclient: build_initial_monmap 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 10 monclient: monmap: 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:03.377 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 1 librados: starting msgr at 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 1 librados: starting objecter 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 1 Processor -- start 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 1 -- start start 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 1 librados: setting wanted keys 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 1 librados: calling monclient init 2024-03-29T11:25:03.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.390+0000 7f0907fff640 10 monclient: init 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient(hunting): picked mon.noname-a con 0x7f09080a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient(hunting): picked mon.noname-b con 0x7f09080aad20 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient(hunting): picked mon.noname-c con 0x7f09080add40 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090806d8c0 con 0x7f09080a9750 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090806db10 con 0x7f09080aad20 2024-03-29T11:25:03.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090813ee30 con 0x7f09080add40 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f0907fff640 10 monclient(hunting): authenticate will time out at 1223.654551s 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f09077fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f09080add40 0x7f09081db750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:59584/0 (socket says 172.21.15.19:59584) 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f09077fe640 1 -- 172.21.15.19:0/2268117979 learned_addr learned my addr 172.21.15.19:0/2268117979 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.391+0000 7f08ef7fe640 10 client.?.objecter ms_handle_connect 0x7f09080add40 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422175285 0 0) 0x7f090813ee30 con 0x7f09080add40 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:25:03.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f4004eb0 con 0x7f09080add40 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294829320 0 0) 0x7f08f4004eb0 con 0x7f09080add40 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.392+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f090813ee30 con 0x7f09080add40 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f08f8001650 con 0x7f09080add40 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:03.385 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:03.386 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:03.386 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:03.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:03.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 10 client.?.objecter ms_handle_connect 0x7f09080aad20 2024-03-29T11:25:03.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:03.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1666563965 0 0) 0x7f090813ee30 con 0x7f09080add40 2024-03-29T11:25:03.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 >> v1:172.21.15.62:6789/0 conn(0x7f09080aad20 legacy=0x7f09081db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:03.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.393+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 >> v1:172.21.15.19:6789/0 conn(0x7f09080a9750 legacy=0x7f090806e7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:03.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 1 monclient: found mon.c 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090806eee0 con 0x7f09080add40 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 10 monclient: _renew_subs 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09081e68d0 con 0x7f09080add40 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 10 monclient: _renew_subs 2024-03-29T11:25:03.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09081e6c40 con 0x7f09080add40 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08f8004c70 con 0x7f09080add40 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f0907fff640 1 librados: init done 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f08f8004f80 con 0x7f09080add40 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f08ef7fe640 10 monclient: dump: 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:03.389 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.395+0000 7f09057fa640 4 set_mon_vals no callback set 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.397+0000 7f0907fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.397+0000 7f0907fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.397+0000 7f0907fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.397+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e8005080 con 0x7f09080add40 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.397+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f08f801af50 con 0x7f09080add40 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f08f8084d60 con 0x7f09080add40 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f08ef7fe640 10 client.4448.objecter ms_dispatch 0x7f0908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f08ef7fe640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:03.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f08ef7fe640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f08ef7fe640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:25:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f0906ffd640 10 monclient: get_auth_request con 0x7f08f406dc80 auth_method 0 2024-03-29T11:25:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.398+0000 7f08ef7fe640 10 client.4448.objecter ms_handle_connect 0x7f08f406dc80 2024-03-29T11:25:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.405+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08f8050640 con 0x7f09080add40 2024-03-29T11:25:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.405+0000 7f08ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:03.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.405+0000 7f08ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:03.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.722+0000 7f0907fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:03.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.722+0000 7f0907fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:03.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.722+0000 7f0907fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:03.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.722+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08e8005330 con 0x7f09080add40 2024-03-29T11:25:03.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.723+0000 7f08ef7fe640 1 -- 172.21.15.19:0/2268117979 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f08f8052570 con 0x7f09080add40 2024-03-29T11:25:03.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.723+0000 7f08ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:03.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.723+0000 7f08ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:03.707 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:03.707 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:03.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 10 librados: watch_flush enter 2024-03-29T11:25:03.711 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 10 librados: watch_flush exit 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 >> v1:172.21.15.19:6816/3721232097 conn(0x7f08f406dc80 legacy=0x7f08f4070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 10 monclient: shutdown 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.727+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 >> v1:172.21.15.19:6790/0 conn(0x7f09080add40 legacy=0x7f09081db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.728+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 shutdown_connections 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.728+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 >> 172.21.15.19:0/2268117979 conn(0x7f09080673e0 msgr2=0x7f09081404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.728+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 shutdown_connections 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.729+0000 7f0907fff640 1 -- 172.21.15.19:0/2268117979 wait complete. 2024-03-29T11:25:03.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:03.729+0000 7f0907fff640 1 librados: shutdown 2024-03-29T11:25:03.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:10.533 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:10.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.695+0000 7f8e0b255640 10 monclient: get_monmap_and_config 2024-03-29T11:25:10.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.695+0000 7f8e0b255640 10 monclient: build_initial_monmap 2024-03-29T11:25:10.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.695+0000 7f8e0b255640 10 monclient: monmap: 2024-03-29T11:25:10.678 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:10.678 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:10.696080+0000 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:10.696080+0000 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:10.679 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.696+0000 7f8e0b255640 1 Processor -- start 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.696+0000 7f8e0b255640 1 -- start start 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.696+0000 7f8e0b255640 10 monclient: init 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient(hunting): picked mon.noname-a con 0x7f8e041422b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient(hunting): picked mon.noname-b con 0x7f8e04142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient(hunting): picked mon.noname-c con 0x7f8e040b17d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e04142c80 con 0x7f8e041422b0 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e040aae40 con 0x7f8e040b17d0 2024-03-29T11:25:10.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e0413fbd0 con 0x7f8e04142690 2024-03-29T11:25:10.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:10.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e0b255640 10 monclient(hunting): authenticate will time out at 1230.960538s 2024-03-29T11:25:10.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e03fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f8e04142690 0x7f8e040b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:55086/0 (socket says 172.21.15.19:55086) 2024-03-29T11:25:10.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.697+0000 7f8e03fff640 1 -- 172.21.15.19:0/3476356493 learned_addr learned my addr 172.21.15.19:0/3476356493 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207744977 0 0) 0x7f8e040aae40 con 0x7f8e040b17d0 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df4003520 con 0x7f8e040b17d0 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733714666 0 0) 0x7f8e0413fbd0 con 0x7f8e04142690 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:25:10.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.698+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e040aae40 con 0x7f8e04142690 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.699+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3555354123 0 0) 0x7f8e040aae40 con 0x7f8e04142690 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.699+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 >> v1:172.21.15.62:6789/0 conn(0x7f8e040b17d0 legacy=0x7f8e040b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.699+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 >> v1:172.21.15.19:6789/0 conn(0x7f8e041422b0 legacy=0x7f8e0413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:10.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04151940 con 0x7f8e04142690 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8df0001670 con 0x7f8e04142690 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e037fe640 10 monclient: dump: 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:10.685 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e0b255640 5 monclient: authenticate success, global_id 4451 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.700+0000 7f8e0b255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8df00017d0 con 0x7f8e04142690 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:10.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e037fe640 1 -- 172.21.15.19:0/3476356493 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8df0001b60 con 0x7f8e04142690 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e037fe640 10 monclient: dump: 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:10.687 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:10.688 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e0b255640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:10.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.701+0000 7f8e0b255640 4 set_mon_vals no callback set 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.702+0000 7f8e0b255640 10 monclient: shutdown 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.702+0000 7f8e0b255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.702+0000 7f8e0b255640 1 -- 172.21.15.19:0/3476356493 >> v1:172.21.15.19:6790/0 conn(0x7f8e04142690 legacy=0x7f8e040b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.702+0000 7f8e0b255640 1 -- 172.21.15.19:0/3476356493 shutdown_connections 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.702+0000 7f8e0b255640 1 -- 172.21.15.19:0/3476356493 >> 172.21.15.19:0/3476356493 conn(0x7f8e04066a00 msgr2=0x7f8e04066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.703+0000 7f8e0b255640 1 -- 172.21.15.19:0/3476356493 shutdown_connections 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.703+0000 7f8e0b255640 1 -- 172.21.15.19:0/3476356493 wait complete. 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.703+0000 7f8e0b255640 10 monclient: build_initial_monmap 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.703+0000 7f8e0b255640 10 monclient: monmap: 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:10.691 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 1 librados: starting msgr at 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 1 librados: starting objecter 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 1 Processor -- start 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 1 -- start start 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 1 librados: setting wanted keys 2024-03-29T11:25:10.692 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 1 librados: calling monclient init 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.704+0000 7f8e0b255640 10 monclient: init 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.705+0000 7f8e0b255640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.705+0000 7f8e0b255640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.705+0000 7f8e0b255640 10 monclient(hunting): picked mon.noname-b con 0x7f8e040b17d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.705+0000 7f8e0b255640 10 monclient(hunting): picked mon.noname-c con 0x7f8e041422b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.705+0000 7f8e03fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f8e041422b0 0x7f8e040ab540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:56192/0 (socket says 172.21.15.19:56192) 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.706+0000 7f8e08ff1640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f8e040b17d0 0x7f8e040b0970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:35210/0 (socket says 172.21.15.19:35210) 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.706+0000 7f8e0b255640 10 monclient(hunting): picked mon.noname-a con 0x7f8e04142690 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.706+0000 7f8e03fff640 1 -- 172.21.15.19:0/3526726798 learned_addr learned my addr 172.21.15.19:0/3526726798 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.706+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e040ac730 con 0x7f8e04142690 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.706+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e040aca10 con 0x7f8e040b17d0 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e040accf0 con 0x7f8e041422b0 2024-03-29T11:25:10.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e0b255640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e0b255640 10 monclient(hunting): authenticate will time out at 1230.970018s 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 client.?.objecter ms_handle_connect 0x7f8e041422b0 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 client.?.objecter ms_handle_connect 0x7f8e040b17d0 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305413708 0 0) 0x7f8e040accf0 con 0x7f8e041422b0 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8de8004eb0 con 0x7f8e041422b0 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285311979 0 0) 0x7f8e040aca10 con 0x7f8e040b17d0 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:10.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:10.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:10.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:25:10.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e040accf0 con 0x7f8e040b17d0 2024-03-29T11:25:10.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 254978327 0 0) 0x7f8de8004eb0 con 0x7f8e041422b0 2024-03-29T11:25:10.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e040aca10 con 0x7f8e041422b0 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8df0001ba0 con 0x7f8e041422b0 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): dump: 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.698 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 231443098 0 0) 0x7f8e040accf0 con 0x7f8e040b17d0 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8de8004eb0 con 0x7f8e040b17d0 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8df8001710 con 0x7f8e040b17d0 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 10 monclient(hunting): dump: 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:10.699 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 36904119 0 0) 0x7f8e040aca10 con 0x7f8e041422b0 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 >> v1:172.21.15.62:6789/0 conn(0x7f8e040b17d0 legacy=0x7f8e040b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:10.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.707+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 >> v1:172.21.15.19:6789/0 conn(0x7f8e04142690 legacy=0x7f8e0412fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.708+0000 7f8e017fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.708+0000 7f8e017fa640 1 monclient: found mon.c 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.708+0000 7f8e017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.708+0000 7f8e017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.708+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0414e600 con 0x7f8e041422b0 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.708+0000 7f8e017fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:10.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8df0004a90 con 0x7f8e041422b0 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 10 monclient: _renew_subs 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e04132810 con 0x7f8e041422b0 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 10 monclient: _renew_subs 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e040abc50 con 0x7f8e041422b0 2024-03-29T11:25:10.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e0b255640 1 librados: init done 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8df0001d80 con 0x7f8e041422b0 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e017fa640 10 monclient: dump: 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:10.706 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.709+0000 7f8e02ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.711+0000 7f8e0b255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.711+0000 7f8e0b255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.711+0000 7f8e0b255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.711+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de4005080 con 0x7f8e041422b0 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.712+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f8df001acd0 con 0x7f8e041422b0 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.712+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f8df0084bc0 con 0x7f8e041422b0 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.712+0000 7f8e017fa640 10 client.4454.objecter ms_dispatch 0x7f8e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.712+0000 7f8e017fa640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:10.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.712+0000 7f8e017fa640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:10.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.712+0000 7f8e017fa640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:25:10.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.713+0000 7f8e08ff1640 10 monclient: get_auth_request con 0x7f8de806df00 auth_method 0 2024-03-29T11:25:10.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.714+0000 7f8e017fa640 10 client.4454.objecter ms_handle_connect 0x7f8de806df00 2024-03-29T11:25:10.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.720+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8df00504a0 con 0x7f8e041422b0 2024-03-29T11:25:10.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.720+0000 7f8e017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:10.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:10.720+0000 7f8e017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:11.010 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.027+0000 7f8e0b255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:11.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.027+0000 7f8e0b255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:11.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.027+0000 7f8e0b255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:11.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.027+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8de4005330 con 0x7f8e041422b0 2024-03-29T11:25:11.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.028+0000 7f8e017fa640 1 -- 172.21.15.19:0/3526726798 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f8df00523d0 con 0x7f8e041422b0 2024-03-29T11:25:11.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.028+0000 7f8e017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:11.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.028+0000 7f8e017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:11.012 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:11.012 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.031+0000 7f8e0b255640 10 librados: watch_flush enter 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.031+0000 7f8e0b255640 10 librados: watch_flush exit 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.031+0000 7f8e0b255640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.031+0000 7f8e0b255640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 >> v1:172.21.15.19:6816/3721232097 conn(0x7f8de806df00 legacy=0x7f8de80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 10 monclient: shutdown 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:11.015 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 >> v1:172.21.15.19:6790/0 conn(0x7f8e041422b0 legacy=0x7f8e040ab540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:11.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 shutdown_connections 2024-03-29T11:25:11.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 >> 172.21.15.19:0/3526726798 conn(0x7f8e04066a00 msgr2=0x7f8e0413c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:11.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 shutdown_connections 2024-03-29T11:25:11.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 -- 172.21.15.19:0/3526726798 wait complete. 2024-03-29T11:25:11.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:11.032+0000 7f8e0b255640 1 librados: shutdown 2024-03-29T11:25:11.029 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:17.836 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:17.993 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.009+0000 7f7aa2653640 10 monclient: get_monmap_and_config 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.009+0000 7f7aa2653640 10 monclient: build_initial_monmap 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.009+0000 7f7aa2653640 10 monclient: monmap: 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:18.009897+0000 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:18.009897+0000 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:17.994 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:17.995 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:17.995 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:17.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.011+0000 7f7aa2653640 1 Processor -- start 2024-03-29T11:25:17.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.011+0000 7f7aa2653640 1 -- start start 2024-03-29T11:25:17.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.011+0000 7f7aa2653640 10 monclient: init 2024-03-29T11:25:17.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:17.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:17.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient(hunting): picked mon.noname-a con 0x7f7a9c071f40 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:17.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9c0adcb0 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:17.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient(hunting): picked mon.noname-c con 0x7f7a9c13f910 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:17.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c0af000 con 0x7f7a9c071f40 2024-03-29T11:25:17.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c0ab060 con 0x7f7a9c13f910 2024-03-29T11:25:17.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c0aec40 con 0x7f7a9c0adcb0 2024-03-29T11:25:17.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:17.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.012+0000 7f7aa2653640 10 monclient(hunting): authenticate will time out at 1238.275441s 2024-03-29T11:25:18.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7aa1651640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f7a9c071f40 0x7f7a9c13f200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:54069/0 (socket says 172.21.15.19:54069) 2024-03-29T11:25:18.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7aa1651640 1 -- 172.21.15.19:0/315812235 learned_addr learned my addr 172.21.15.19:0/315812235 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:18.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302878941 0 0) 0x7f7a9c0af000 con 0x7f7a9c071f40 2024-03-29T11:25:18.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7a93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:18.006 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7a93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:18.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:18.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7a93fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:25:18.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.017+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a88003520 con 0x7f7a9c071f40 2024-03-29T11:25:18.007 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910971419 0 0) 0x7f7a88003520 con 0x7f7a9c071f40 2024-03-29T11:25:18.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 >> v1:172.21.15.19:6790/0 conn(0x7f7a9c0adcb0 legacy=0x7f7a9c145560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:18.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 >> v1:172.21.15.62:6789/0 conn(0x7f7a9c13f910 legacy=0x7f7a9c145c70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:25:18.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:18.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 1 monclient: found mon.noname-a 2024-03-29T11:25:18.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:18.008 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c0b0030 con 0x7f7a9c071f40 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7a98001670 con 0x7f7a9c071f40 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.018+0000 7f7a93fff640 10 monclient: dump: 2024-03-29T11:25:18.009 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:18.010 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7aa2653640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7aa2653640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a980018d0 con 0x7f7a9c071f40 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7a93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7a93fff640 1 -- 172.21.15.19:0/315812235 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7a98001d20 con 0x7f7a9c071f40 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7a93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7a93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.019+0000 7f7a93fff640 10 monclient: dump: 2024-03-29T11:25:18.011 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:18.012 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:18.012 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.012 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.012 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:18.012 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.020+0000 7f7aa2653640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.020+0000 7f7aa2653640 4 set_mon_vals no callback set 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.021+0000 7f7aa2653640 10 monclient: shutdown 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.021+0000 7f7aa2653640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.021+0000 7f7aa2653640 1 -- 172.21.15.19:0/315812235 >> v1:172.21.15.19:6789/0 conn(0x7f7a9c071f40 legacy=0x7f7a9c13f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.021+0000 7f7aa2653640 1 -- 172.21.15.19:0/315812235 shutdown_connections 2024-03-29T11:25:18.013 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.021+0000 7f7aa2653640 1 -- 172.21.15.19:0/315812235 >> 172.21.15.19:0/315812235 conn(0x7f7a9c0673e0 msgr2=0x7f7a9c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:18.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.024+0000 7f7aa2653640 1 -- 172.21.15.19:0/315812235 shutdown_connections 2024-03-29T11:25:18.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.024+0000 7f7aa2653640 1 -- 172.21.15.19:0/315812235 wait complete. 2024-03-29T11:25:18.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.025+0000 7f7aa2653640 10 monclient: build_initial_monmap 2024-03-29T11:25:18.016 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.025+0000 7f7aa2653640 10 monclient: monmap: 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:18.017 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 1 librados: starting msgr at 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 1 librados: starting objecter 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 1 Processor -- start 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 1 -- start start 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 1 librados: setting wanted keys 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 1 librados: calling monclient init 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.026+0000 7f7aa2653640 10 monclient: init 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.027+0000 7f7aa2653640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.027+0000 7f7aa2653640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.027+0000 7f7aa2653640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9c071f40 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:18.018 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.031+0000 7f7aa2653640 10 monclient(hunting): picked mon.noname-c con 0x7f7a9c0adcb0 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:18.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa2653640 10 monclient(hunting): picked mon.noname-a con 0x7f7a9c13f910 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:18.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa1e52640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f7a9c13f910 0x7f7a9c1d68e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:54225/0 (socket says 172.21.15.19:54225) 2024-03-29T11:25:18.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa1e52640 1 -- 172.21.15.19:0/2247720924 learned_addr learned my addr 172.21.15.19:0/2247720924 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa2653640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c13e540 con 0x7f7a9c13f910 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7a91ffb640 10 client.?.objecter ms_handle_connect 0x7f7a9c13f910 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7a91ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa2653640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c138b10 con 0x7f7a9c071f40 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa2653640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c136be0 con 0x7f7a9c0adcb0 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7aa2653640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7aa2653640 10 monclient(hunting): authenticate will time out at 1238.298046s 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.034+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114014941 0 0) 0x7f7a9c13e540 con 0x7f7a9c13f910 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a80004eb0 con 0x7f7a9c13f910 2024-03-29T11:25:18.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 10 client.?.objecter ms_handle_connect 0x7f7a9c071f40 2024-03-29T11:25:18.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.035+0000 7f7a91ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022008112 0 0) 0x7f7a80004eb0 con 0x7f7a9c13f910 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a9c13e540 con 0x7f7a9c13f910 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167575704 0 0) 0x7f7a9c138b10 con 0x7f7a9c071f40 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:18.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a80004eb0 con 0x7f7a9c071f40 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7a94001ad0 con 0x7f7a9c13f910 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:18.026 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.038+0000 7f7a91ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 10 client.?.objecter ms_handle_connect 0x7f7a9c0adcb0 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1522392554 0 0) 0x7f7a80004eb0 con 0x7f7a9c071f40 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a9c138b10 con 0x7f7a9c071f40 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7a98001d20 con 0x7f7a9c071f40 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.039+0000 7f7a91ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.027 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2867337132 0 0) 0x7f7a9c13e540 con 0x7f7a9c13f910 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 >> v1:172.21.15.19:6790/0 conn(0x7f7a9c0adcb0 legacy=0x7f7a9c1d61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 >> v1:172.21.15.62:6789/0 conn(0x7f7a9c071f40 legacy=0x7f7a9c138400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 1 monclient: found mon.a 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:18.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c137aa0 con 0x7f7a9c13f910 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7a91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 10 monclient: _renew_subs 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c1d6ff0 con 0x7f7a9c13f910 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 10 monclient: _renew_subs 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c1395a0 con 0x7f7a9c13f910 2024-03-29T11:25:18.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.040+0000 7f7aa2653640 1 librados: init done 2024-03-29T11:25:18.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a94002060 con 0x7f7a9c13f910 2024-03-29T11:25:18.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:18.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7a94004d80 con 0x7f7a9c13f910 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 10 monclient: dump: 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:18.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.042+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f7a94004f20 con 0x7f7a9c13f910 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f7a94084e20 con 0x7f7a9c13f910 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a91ffb640 10 client.4440.objecter ms_dispatch 0x7f7a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a91ffb640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a91ffb640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a91ffb640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a877fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a877fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.043+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9c04f730 con 0x7f7a9c13f910 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.045+0000 7f7a937fe640 4 set_mon_vals no callback set 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.046+0000 7f7aa1651640 10 monclient: get_auth_request con 0x7f7a8006def0 auth_method 0 2024-03-29T11:25:18.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.046+0000 7f7a91ffb640 10 client.4440.objecter ms_handle_connect 0x7f7a8006def0 2024-03-29T11:25:18.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.054+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a94050780 con 0x7f7a9c13f910 2024-03-29T11:25:18.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.054+0000 7f7a91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:18.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.054+0000 7f7a91ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:18.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.399+0000 7f7a877fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:18.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.399+0000 7f7a877fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:18.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.399+0000 7f7a877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:18.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.399+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a9c1398a0 con 0x7f7a9c13f910 2024-03-29T11:25:18.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.401+0000 7f7a91ffb640 1 -- 172.21.15.19:0/2247720924 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f7a940526b0 con 0x7f7a9c13f910 2024-03-29T11:25:18.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.401+0000 7f7a91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:18.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.401+0000 7f7a91ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:18.387 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:18.387 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 10 librados: watch_flush enter 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 10 librados: watch_flush exit 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 10 client.4440.objecter successfully canceled tick 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 >> v1:172.21.15.19:6816/3721232097 conn(0x7f7a8006def0 legacy=0x7f7a800703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 10 monclient: shutdown 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:18.391 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 >> v1:172.21.15.19:6789/0 conn(0x7f7a9c13f910 legacy=0x7f7a9c1d68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:18.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 shutdown_connections 2024-03-29T11:25:18.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.407+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 >> 172.21.15.19:0/2247720924 conn(0x7f7a9c0673e0 msgr2=0x7f7a9c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:18.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.408+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 shutdown_connections 2024-03-29T11:25:18.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.408+0000 7f7a877fe640 1 -- 172.21.15.19:0/2247720924 wait complete. 2024-03-29T11:25:18.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:18.408+0000 7f7a877fe640 1 librados: shutdown 2024-03-29T11:25:18.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:25.217 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:25.369 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.386+0000 7f01fea20640 10 monclient: get_monmap_and_config 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.386+0000 7f01fea20640 10 monclient: build_initial_monmap 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.386+0000 7f01fea20640 10 monclient: monmap: 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:25.386836+0000 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:25.386836+0000 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:25.370 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:25.371 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:25.371 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.387+0000 7f01fea20640 1 Processor -- start 2024-03-29T11:25:25.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.389+0000 7f01fea20640 1 -- start start 2024-03-29T11:25:25.373 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.389+0000 7f01fea20640 10 monclient: init 2024-03-29T11:25:25.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.389+0000 7f01fea20640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:25.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.389+0000 7f01fea20640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:25.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.390+0000 7f01fea20640 10 monclient(hunting): picked mon.noname-a con 0x7f01f81457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:25.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.390+0000 7f01fea20640 10 monclient(hunting): picked mon.noname-c con 0x7f01f813f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:25.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fea20640 10 monclient(hunting): picked mon.noname-b con 0x7f01f80a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:25.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fcfbd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f01f80a9d70 0x7f01f814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:41327/0 (socket says 172.21.15.19:41327) 2024-03-29T11:25:25.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fcfbd640 1 -- 172.21.15.19:0/1105039904 learned_addr learned my addr 172.21.15.19:0/1105039904 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:25.377 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fea20640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f80aaef0 con 0x7f01f81457f0 2024-03-29T11:25:25.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8145bd0 con 0x7f01f813f650 2024-03-29T11:25:25.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8145eb0 con 0x7f01f80a9d70 2024-03-29T11:25:25.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fea20640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:25.378 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.393+0000 7f01fea20640 10 monclient(hunting): authenticate will time out at 1245.656969s 2024-03-29T11:25:25.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.394+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779225826 0 0) 0x7f01f8145eb0 con 0x7f01f80a9d70 2024-03-29T11:25:25.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:25.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e0003520 con 0x7f01f80a9d70 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103108838 0 0) 0x7f01f80aaef0 con 0x7f01f81457f0 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f8145eb0 con 0x7f01f81457f0 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 96606562 0 0) 0x7f01f8145eb0 con 0x7f01f81457f0 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 >> v1:172.21.15.19:6790/0 conn(0x7f01f80a9d70 legacy=0x7f01f814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.395+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 >> v1:172.21.15.62:6789/0 conn(0x7f01f813f650 legacy=0x7f01f80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:25.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.396+0000 7f01f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f8146340 con 0x7f01f81457f0 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01fea20640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:25:25.383 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01fea20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f01ec001670 con 0x7f01f81457f0 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01f6ffd640 10 monclient: dump: 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:25.384 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:25.385 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:25.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.397+0000 7f01fea20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:25.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01ec001890 con 0x7f01f81457f0 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01f6ffd640 1 -- 172.21.15.19:0/1105039904 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f01ec001c20 con 0x7f01f81457f0 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01f6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01f6ffd640 10 monclient: dump: 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.386 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01fea20640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.399+0000 7f01fea20640 4 set_mon_vals no callback set 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.400+0000 7f01fea20640 10 monclient: shutdown 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.400+0000 7f01fea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.400+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 >> v1:172.21.15.19:6789/0 conn(0x7f01f81457f0 legacy=0x7f01f813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.400+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 shutdown_connections 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.400+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 >> 172.21.15.19:0/1105039904 conn(0x7f01f8066a00 msgr2=0x7f01f8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.401+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 shutdown_connections 2024-03-29T11:25:25.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.401+0000 7f01fea20640 1 -- 172.21.15.19:0/1105039904 wait complete. 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.402+0000 7f01fea20640 10 monclient: build_initial_monmap 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.402+0000 7f01fea20640 10 monclient: monmap: 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:25.389 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.402+0000 7f01fea20640 1 librados: starting msgr at 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.402+0000 7f01fea20640 1 librados: starting objecter 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.402+0000 7f01fea20640 1 Processor -- start 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.403+0000 7f01fea20640 1 -- start start 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.403+0000 7f01fea20640 1 librados: setting wanted keys 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.403+0000 7f01fea20640 1 librados: calling monclient init 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.403+0000 7f01fea20640 10 monclient: init 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.403+0000 7f01fea20640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.403+0000 7f01fea20640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.404+0000 7f01fea20640 10 monclient(hunting): picked mon.noname-c con 0x7f01f80a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:25.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.404+0000 7f01fea20640 10 monclient(hunting): picked mon.noname-b con 0x7f01f813f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.407+0000 7f01fea20640 10 monclient(hunting): picked mon.noname-a con 0x7f01f81457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.407+0000 7f01fea20640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f81db830 con 0x7f01f81457f0 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.407+0000 7f01fea20640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f8140350 con 0x7f01f813f650 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.407+0000 7f01fcfbd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f01f81457f0 0x7f01f81e7070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:52911/0 (socket says 172.21.15.19:52911) 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.407+0000 7f01fcfbd640 1 -- 172.21.15.19:0/2553525447 learned_addr learned my addr 172.21.15.19:0/2553525447 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.408+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f813fda0 con 0x7f01f80a9d70 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.408+0000 7f01fea20640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:25.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.408+0000 7f01fea20640 10 monclient(hunting): authenticate will time out at 1245.671219s 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.411+0000 7f01f4ff9640 10 client.?.objecter ms_handle_connect 0x7f01f81457f0 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.411+0000 7f01f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.411+0000 7f01f4ff9640 10 client.?.objecter ms_handle_connect 0x7f01f80a9d70 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.411+0000 7f01f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292296494 0 0) 0x7f01f81db830 con 0x7f01f81457f0 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:25.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e4004eb0 con 0x7f01f81457f0 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979629458 0 0) 0x7f01f813fda0 con 0x7f01f80a9d70 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.412+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f81db830 con 0x7f01f80a9d70 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.413+0000 7f01f4ff9640 10 client.?.objecter ms_handle_connect 0x7f01f813f650 2024-03-29T11:25:25.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.413+0000 7f01f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569950190 0 0) 0x7f01f8140350 con 0x7f01f813f650 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f813fda0 con 0x7f01f813f650 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470223485 0 0) 0x7f01f81db830 con 0x7f01f80a9d70 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f8140350 con 0x7f01f80a9d70 2024-03-29T11:25:25.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f01ec001ba0 con 0x7f01f80a9d70 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:25.400 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3970569862 0 0) 0x7f01f8140350 con 0x7f01f80a9d70 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 >> v1:172.21.15.62:6789/0 conn(0x7f01f813f650 legacy=0x7f01f8142290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 >> v1:172.21.15.19:6789/0 conn(0x7f01f81457f0 legacy=0x7f01f81e7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 monclient: found mon.c 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:25.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f813ff30 con 0x7f01f80a9d70 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 10 monclient: _renew_subs 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f813dfa0 con 0x7f01f80a9d70 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 10 monclient: _renew_subs 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f81dc300 con 0x7f01f80a9d70 2024-03-29T11:25:25.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.414+0000 7f01fea20640 1 librados: init done 2024-03-29T11:25:25.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01ec001d60 con 0x7f01f80a9d70 2024-03-29T11:25:25.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:25.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f01ec003a40 con 0x7f01f80a9d70 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f4ff9640 10 monclient: dump: 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:25.405 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.415+0000 7f01f67fc640 4 set_mon_vals no callback set 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.418+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f01ec019a10 con 0x7f01f80a9d70 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f01ec083890 con 0x7f01f80a9d70 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f4ff9640 10 client.4460.objecter ms_dispatch 0x7f01f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f4ff9640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f4ff9640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f77fe640 10 monclient: get_auth_request con 0x7f01e406dea0 auth_method 0 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f4ff9640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01fea20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01f4ff9640 10 client.4460.objecter ms_handle_connect 0x7f01e406dea0 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.419+0000 7f01fea20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.420+0000 7f01fea20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:25.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.420+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d8005080 con 0x7f01f80a9d70 2024-03-29T11:25:25.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.428+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01ec04f170 con 0x7f01f80a9d70 2024-03-29T11:25:25.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.428+0000 7f01f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:25.411 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.428+0000 7f01f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:25.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.757+0000 7f01fea20640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:25.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.757+0000 7f01fea20640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:25.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.757+0000 7f01fea20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:25.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.757+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d8005330 con 0x7f01f80a9d70 2024-03-29T11:25:25.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.758+0000 7f01f4ff9640 1 -- 172.21.15.19:0/2553525447 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f01ec0510a0 con 0x7f01f80a9d70 2024-03-29T11:25:25.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.758+0000 7f01f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:25.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.758+0000 7f01f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:25.743 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:25.743 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:25.746 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 10 librados: watch_flush enter 2024-03-29T11:25:25.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 10 librados: watch_flush exit 2024-03-29T11:25:25.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T11:25:25.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 10 client.4460.objecter successfully canceled tick 2024-03-29T11:25:25.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 >> v1:172.21.15.19:6816/3721232097 conn(0x7f01e406dea0 legacy=0x7f01e4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:25.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 10 monclient: shutdown 2024-03-29T11:25:25.747 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:25.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.763+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 >> v1:172.21.15.19:6790/0 conn(0x7f01f80a9d70 legacy=0x7f01f81dbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:25.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.764+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 shutdown_connections 2024-03-29T11:25:25.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.764+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 >> 172.21.15.19:0/2553525447 conn(0x7f01f8066a00 msgr2=0x7f01f80b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:25.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.764+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 shutdown_connections 2024-03-29T11:25:25.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.764+0000 7f01fea20640 1 -- 172.21.15.19:0/2553525447 wait complete. 2024-03-29T11:25:25.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:25.764+0000 7f01fea20640 1 librados: shutdown 2024-03-29T11:25:25.765 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:32.573 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:32.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.737+0000 7fc8235c1640 10 monclient: get_monmap_and_config 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.737+0000 7fc8235c1640 10 monclient: build_initial_monmap 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.737+0000 7fc8235c1640 10 monclient: monmap: 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:32.738566+0000 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:32.738566+0000 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:32.722 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:32.723 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:32.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.738+0000 7fc8235c1640 1 Processor -- start 2024-03-29T11:25:32.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.738+0000 7fc8235c1640 1 -- start start 2024-03-29T11:25:32.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.738+0000 7fc8235c1640 10 monclient: init 2024-03-29T11:25:32.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:32.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:32.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc81c1457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc81c13f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc81c0a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c0aaef0 con 0x7fc81c13f650 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c145bd0 con 0x7fc81c0a9d70 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.739+0000 7fc8235c1640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c145eb0 con 0x7fc81c1457f0 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.740+0000 7fc8235c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.740+0000 7fc8235c1640 10 monclient(hunting): authenticate will time out at 1253.003007s 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.743+0000 7fc821b5e640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc81c0a9d70 0x7fc81c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:32809/0 (socket says 172.21.15.19:32809) 2024-03-29T11:25:32.728 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.743+0000 7fc821b5e640 1 -- 172.21.15.19:0/3450250731 learned_addr learned my addr 172.21.15.19:0/3450250731 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344153287 0 0) 0x7fc81c145bd0 con 0x7fc81c0a9d70 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc808003520 con 0x7fc81c0a9d70 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.745+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 268706645 0 0) 0x7fc808003520 con 0x7fc81c0a9d70 2024-03-29T11:25:32.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 >> v1:172.21.15.19:6790/0 conn(0x7fc81c1457f0 legacy=0x7fc81c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:32.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 >> v1:172.21.15.19:6789/0 conn(0x7fc81c13f650 legacy=0x7fc81c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:32.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:32.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 1 monclient: found mon.noname-c 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c146340 con 0x7fc81c0a9d70 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc814001a10 con 0x7fc81c0a9d70 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.746+0000 7fc813fff640 10 monclient: dump: 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.734 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc814001be0 con 0x7fc81c0a9d70 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc813fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc813fff640 1 -- 172.21.15.19:0/3450250731 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc814002030 con 0x7fc81c0a9d70 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc813fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc813fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc813fff640 10 monclient: dump: 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:32.735 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:32.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc8235c1640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc8235c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.747+0000 7fc8235c1640 4 set_mon_vals no callback set 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.748+0000 7fc8235c1640 10 monclient: shutdown 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.748+0000 7fc8235c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.748+0000 7fc8235c1640 1 -- 172.21.15.19:0/3450250731 >> v1:172.21.15.62:6789/0 conn(0x7fc81c0a9d70 legacy=0x7fc81c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.750+0000 7fc8235c1640 1 -- 172.21.15.19:0/3450250731 shutdown_connections 2024-03-29T11:25:32.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.750+0000 7fc8235c1640 1 -- 172.21.15.19:0/3450250731 >> 172.21.15.19:0/3450250731 conn(0x7fc81c066a00 msgr2=0x7fc81c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:32.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.755+0000 7fc8235c1640 1 -- 172.21.15.19:0/3450250731 shutdown_connections 2024-03-29T11:25:32.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.757+0000 7fc8235c1640 1 -- 172.21.15.19:0/3450250731 wait complete. 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 10 monclient: build_initial_monmap 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 10 monclient: monmap: 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:32.748 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 1 librados: starting msgr at 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 1 librados: starting objecter 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 1 Processor -- start 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 1 -- start start 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 1 librados: setting wanted keys 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 1 librados: calling monclient init 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.758+0000 7fc8235c1640 10 monclient: init 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc81c0a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc81c13f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:32.749 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc81c1457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c135f00 con 0x7fc81c1457f0 2024-03-29T11:25:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c070cc0 con 0x7fc81c0a9d70 2024-03-29T11:25:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc81c071a50 con 0x7fc81c13f650 2024-03-29T11:25:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:32.750 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.759+0000 7fc8235c1640 10 monclient(hunting): authenticate will time out at 1253.022622s 2024-03-29T11:25:32.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc820b5c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fc81c13f650 0x7fc81c1350e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48729/0 (socket says 172.21.15.19:48729) 2024-03-29T11:25:32.751 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc820b5c640 1 -- 172.21.15.19:0/2638146356 learned_addr learned my addr 172.21.15.19:0/2638146356 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 10 client.?.objecter ms_handle_connect 0x7fc81c13f650 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220473389 0 0) 0x7fc81c071a50 con 0x7fc81c13f650 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.763+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc808003520 con 0x7fc81c13f650 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3140082241 0 0) 0x7fc808003520 con 0x7fc81c13f650 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc81c071a50 con 0x7fc81c13f650 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc80c004080 con 0x7fc81c13f650 2024-03-29T11:25:32.752 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:32.753 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1156903604 0 0) 0x7fc81c071a50 con 0x7fc81c13f650 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.764+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 >> v1:172.21.15.62:6789/0 conn(0x7fc81c0a9d70 legacy=0x7fc81c1de960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 >> v1:172.21.15.19:6789/0 conn(0x7fc81c1457f0 legacy=0x7fc81c1357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 1 monclient: found mon.c 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c06f660 con 0x7fc81c13f650 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 client.?.objecter ms_handle_connect 0x7fc81c1457f0 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:25:32.754 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 10 monclient: _renew_subs 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81c06f8d0 con 0x7fc81c13f650 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 10 monclient: _renew_subs 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81c0703f0 con 0x7fc81c13f650 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc8235c1640 1 librados: init done 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc80c0043a0 con 0x7fc81c13f650 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc80c001320 con 0x7fc81c13f650 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:32.755 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.765+0000 7fc811ffb640 10 monclient: dump: 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:32.756 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.766+0000 7fc8137fe640 4 set_mon_vals no callback set 2024-03-29T11:25:32.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.767+0000 7fc8235c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:32.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.767+0000 7fc8235c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:32.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.767+0000 7fc8235c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:32.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.767+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7fc005080 con 0x7fc81c13f650 2024-03-29T11:25:32.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.768+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fc80c001480 con 0x7fc81c13f650 2024-03-29T11:25:32.758 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.769+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fc80c0848f0 con 0x7fc81c13f650 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.769+0000 7fc811ffb640 10 client.4463.objecter ms_dispatch 0x7fc81c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.769+0000 7fc811ffb640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.769+0000 7fc811ffb640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.769+0000 7fc811ffb640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.770+0000 7fc82135d640 10 monclient: get_auth_request con 0x7fc80806df60 auth_method 0 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.771+0000 7fc811ffb640 10 client.4463.objecter ms_handle_connect 0x7fc80806df60 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.774+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc80c0501d0 con 0x7fc81c13f650 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.774+0000 7fc811ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:32.759 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:32.774+0000 7fc811ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:33.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.096+0000 7fc8235c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:33.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.096+0000 7fc8235c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:33.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.096+0000 7fc8235c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:33.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.096+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7fc005330 con 0x7fc81c13f650 2024-03-29T11:25:33.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.099+0000 7fc811ffb640 1 -- 172.21.15.19:0/2638146356 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fc80c052100 con 0x7fc81c13f650 2024-03-29T11:25:33.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.099+0000 7fc811ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:33.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.099+0000 7fc811ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:33.083 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:33.083 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:33.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.102+0000 7fc8235c1640 10 librados: watch_flush enter 2024-03-29T11:25:33.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.102+0000 7fc8235c1640 10 librados: watch_flush exit 2024-03-29T11:25:33.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.102+0000 7fc8235c1640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:25:33.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.102+0000 7fc8235c1640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:25:33.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 >> v1:172.21.15.19:6816/3721232097 conn(0x7fc80806df60 legacy=0x7fc808070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 10 monclient: shutdown 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 >> v1:172.21.15.19:6790/0 conn(0x7fc81c13f650 legacy=0x7fc81c1350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 shutdown_connections 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 >> 172.21.15.19:0/2638146356 conn(0x7fc81c066a00 msgr2=0x7fc81c0b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 shutdown_connections 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.105+0000 7fc8235c1640 1 -- 172.21.15.19:0/2638146356 wait complete. 2024-03-29T11:25:33.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:33.106+0000 7fc8235c1640 1 librados: shutdown 2024-03-29T11:25:33.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:39.914 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:40.062 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.077+0000 7fd254a33640 10 monclient: get_monmap_and_config 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.077+0000 7fd254a33640 10 monclient: build_initial_monmap 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.077+0000 7fd254a33640 10 monclient: monmap: 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:40.078411+0000 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:40.078411+0000 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:40.063 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.078+0000 7fd254a33640 1 Processor -- start 2024-03-29T11:25:40.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.078+0000 7fd254a33640 1 -- start start 2024-03-29T11:25:40.064 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.078+0000 7fd254a33640 10 monclient: init 2024-03-29T11:25:40.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.079+0000 7fd254a33640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:40.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.079+0000 7fd254a33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:40.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.079+0000 7fd254a33640 10 monclient(hunting): picked mon.noname-b con 0x7fd2500a9750 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:40.065 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 10 monclient(hunting): picked mon.noname-c con 0x7fd2500aacf0 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 10 monclient(hunting): picked mon.noname-a con 0x7fd2500af220 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500ac2f0 con 0x7fd2500af220 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500acaf0 con 0x7fd2500aacf0 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500ac710 con 0x7fd2500a9750 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd254a33640 10 monclient(hunting): authenticate will time out at 1260.343229s 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd247fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fd2500aacf0 0x7fd25013e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:37810/0 (socket says 172.21.15.19:37810) 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.080+0000 7fd247fff640 1 -- 172.21.15.19:0/3803579473 learned_addr learned my addr 172.21.15.19:0/3803579473 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520480588 0 0) 0x7fd2500acaf0 con 0x7fd2500aacf0 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:40.066 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c003520 con 0x7fd2500aacf0 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535268886 0 0) 0x7fd2500ac710 con 0x7fd2500a9750 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:25:40.067 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2500acaf0 con 0x7fd2500a9750 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.081+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1294290291 0 0) 0x7fd2500acaf0 con 0x7fd2500a9750 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 >> v1:172.21.15.62:6789/0 conn(0x7fd2500aacf0 legacy=0x7fd25013e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 >> v1:172.21.15.19:6789/0 conn(0x7fd2500af220 legacy=0x7fd250149f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500ac8a0 con 0x7fd2500a9750 2024-03-29T11:25:40.069 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd240001670 con 0x7fd2500a9750 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd254a33640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd254a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd24dd9b640 10 monclient: dump: 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:40.070 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.082+0000 7fd254a33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2400017d0 con 0x7fd2500a9750 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd24dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:40.071 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd24dd9b640 1 -- 172.21.15.19:0/3803579473 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd240001c20 con 0x7fd2500a9750 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd24dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd24dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd24dd9b640 10 monclient: dump: 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:40.072 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd254a33640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.083+0000 7fd254a33640 4 set_mon_vals no callback set 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 10 monclient: shutdown 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 1 -- 172.21.15.19:0/3803579473 >> v1:172.21.15.19:6790/0 conn(0x7fd2500a9750 legacy=0x7fd2500ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 1 -- 172.21.15.19:0/3803579473 shutdown_connections 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 1 -- 172.21.15.19:0/3803579473 >> 172.21.15.19:0/3803579473 conn(0x7fd25006b660 msgr2=0x7fd25006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 1 -- 172.21.15.19:0/3803579473 shutdown_connections 2024-03-29T11:25:40.073 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.084+0000 7fd254a33640 1 -- 172.21.15.19:0/3803579473 wait complete. 2024-03-29T11:25:40.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.086+0000 7fd254a33640 10 monclient: build_initial_monmap 2024-03-29T11:25:40.080 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.086+0000 7fd254a33640 10 monclient: monmap: 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.086+0000 7fd254a33640 1 librados: starting msgr at 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.086+0000 7fd254a33640 1 librados: starting objecter 2024-03-29T11:25:40.081 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.087+0000 7fd254a33640 1 Processor -- start 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.087+0000 7fd254a33640 1 -- start start 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.087+0000 7fd254a33640 1 librados: setting wanted keys 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.087+0000 7fd254a33640 1 librados: calling monclient init 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.087+0000 7fd254a33640 10 monclient: init 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient(hunting): picked mon.noname-b con 0x7fd2500a9750 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient(hunting): picked mon.noname-c con 0x7fd2500aacf0 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient(hunting): picked mon.noname-a con 0x7fd2500af220 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25013afd0 con 0x7fd2500af220 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd25006f8b0 con 0x7fd2500a9750 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd250135f50 con 0x7fd2500aacf0 2024-03-29T11:25:40.082 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd247fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd2500aacf0 0x7fd25006dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:41154/0 (socket says 172.21.15.19:41154) 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd247fff640 1 -- 172.21.15.19:0/1094234788 learned_addr learned my addr 172.21.15.19:0/1094234788 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd254a33640 10 monclient(hunting): authenticate will time out at 1260.351477s 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd246ffd640 10 client.?.objecter ms_handle_connect 0x7fd2500aacf0 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd246ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd246ffd640 10 client.?.objecter ms_handle_connect 0x7fd2500a9750 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.088+0000 7fd246ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 client.?.objecter ms_handle_connect 0x7fd2500af220 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151717468 0 0) 0x7fd25006f8b0 con 0x7fd2500a9750 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:40.083 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230004eb0 con 0x7fd2500a9750 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972170442 0 0) 0x7fd250135f50 con 0x7fd2500aacf0 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd25006f8b0 con 0x7fd2500aacf0 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.089+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 255076355 0 0) 0x7fd230004eb0 con 0x7fd2500a9750 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd250135f50 con 0x7fd2500a9750 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd240001ae0 con 0x7fd2500a9750 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:40.084 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732565923 0 0) 0x7fd25013afd0 con 0x7fd2500af220 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:40.085 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd230004eb0 con 0x7fd2500af220 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3692072394 0 0) 0x7fd250135f50 con 0x7fd2500a9750 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 >> v1:172.21.15.19:6790/0 conn(0x7fd2500aacf0 legacy=0x7fd25006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 >> v1:172.21.15.19:6789/0 conn(0x7fd2500af220 legacy=0x7fd25013b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 monclient: found mon.b 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2501360e0 con 0x7fd2500a9750 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd246ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:40.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd254a33640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd254a33640 10 monclient: _renew_subs 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd254a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25006e350 con 0x7fd2500a9750 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.090+0000 7fd254a33640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd254a33640 10 monclient: _renew_subs 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd254a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25013bbc0 con 0x7fd2500a9750 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd254a33640 1 librados: init done 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2400038a0 con 0x7fd2500a9750 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd246ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd24d59a640 4 set_mon_vals no callback set 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd240003bb0 con 0x7fd2500a9750 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd246ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:40.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd246ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.091+0000 7fd246ffd640 10 monclient: dump: 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.092+0000 7fd254a33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.092+0000 7fd254a33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.092+0000 7fd254a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:25:40.088 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.093+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd234005080 con 0x7fd2500a9750 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.093+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fd240019b80 con 0x7fd2500a9750 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.094+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fd240083910 con 0x7fd2500a9750 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.094+0000 7fd247fff640 10 monclient: get_auth_request con 0x7fd23006dc00 auth_method 0 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.094+0000 7fd246ffd640 10 client.4441.objecter ms_dispatch 0x7fd250000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.094+0000 7fd246ffd640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.094+0000 7fd246ffd640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.094+0000 7fd246ffd640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.095+0000 7fd246ffd640 10 client.4441.objecter ms_handle_connect 0x7fd23006dc00 2024-03-29T11:25:40.089 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.101+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd24004f1f0 con 0x7fd2500a9750 2024-03-29T11:25:40.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.101+0000 7fd246ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:40.090 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.101+0000 7fd246ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:40.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.454+0000 7fd254a33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:40.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.454+0000 7fd254a33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:40.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.454+0000 7fd254a33640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:25:40.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.454+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd234005330 con 0x7fd2500a9750 2024-03-29T11:25:40.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.455+0000 7fd246ffd640 1 -- 172.21.15.19:0/1094234788 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fd240051120 con 0x7fd2500a9750 2024-03-29T11:25:40.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.455+0000 7fd246ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:40.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.455+0000 7fd246ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:40.440 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:40.440 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:40.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 10 librados: watch_flush enter 2024-03-29T11:25:40.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 10 librados: watch_flush exit 2024-03-29T11:25:40.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T11:25:40.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 10 client.4441.objecter successfully canceled tick 2024-03-29T11:25:40.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 >> v1:172.21.15.19:6816/3721232097 conn(0x7fd23006dc00 legacy=0x7fd2300700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.444 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 10 monclient: shutdown 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 >> v1:172.21.15.62:6789/0 conn(0x7fd2500a9750 legacy=0x7fd250135840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 shutdown_connections 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 >> 172.21.15.19:0/1094234788 conn(0x7fd25006b660 msgr2=0x7fd2500b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 shutdown_connections 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.460+0000 7fd254a33640 1 -- 172.21.15.19:0/1094234788 wait complete. 2024-03-29T11:25:40.445 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:40.461+0000 7fd254a33640 1 librados: shutdown 2024-03-29T11:25:40.460 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:47.267 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:47.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.434+0000 7f43efa27640 10 monclient: get_monmap_and_config 2024-03-29T11:25:47.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.434+0000 7f43efa27640 10 monclient: build_initial_monmap 2024-03-29T11:25:47.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.434+0000 7f43efa27640 10 monclient: monmap: 2024-03-29T11:25:47.419 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:47.420 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:47.420 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:47.435183+0000 2024-03-29T11:25:47.420 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:47.435183+0000 2024-03-29T11:25:47.420 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:47.420 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:47.420 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:47.421 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:47.421 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:47.421 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:47.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.436+0000 7f43efa27640 1 Processor -- start 2024-03-29T11:25:47.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.436+0000 7f43efa27640 1 -- start start 2024-03-29T11:25:47.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.436+0000 7f43efa27640 10 monclient: init 2024-03-29T11:25:47.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:47.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:47.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient(hunting): picked mon.noname-a con 0x7f43e80b1f60 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:47.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient(hunting): picked mon.noname-c con 0x7f43e8071eb0 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:47.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient(hunting): picked mon.noname-b con 0x7f43e8143d10 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:47.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e806ef80 con 0x7f43e80b1f60 2024-03-29T11:25:47.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e806f260 con 0x7f43e8071eb0 2024-03-29T11:25:47.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80b40c0 con 0x7f43e8143d10 2024-03-29T11:25:47.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:47.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.437+0000 7f43efa27640 10 monclient(hunting): authenticate will time out at 1267.700367s 2024-03-29T11:25:47.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.443+0000 7f43ed7c3640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f43e80b1f60 0x7f43e8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48413/0 (socket says 172.21.15.19:48413) 2024-03-29T11:25:47.427 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.443+0000 7f43ed7c3640 1 -- 172.21.15.19:0/139076843 learned_addr learned my addr 172.21.15.19:0/139076843 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:47.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514526439 0 0) 0x7f43e806f260 con 0x7f43e8071eb0 2024-03-29T11:25:47.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:47.430 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:47.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:47.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:25:47.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4003520 con 0x7f43e8071eb0 2024-03-29T11:25:47.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970196672 0 0) 0x7f43e806ef80 con 0x7f43e80b1f60 2024-03-29T11:25:47.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:47.431 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:47.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:47.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:25:47.432 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.445+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e806f260 con 0x7f43e80b1f60 2024-03-29T11:25:47.433 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1783198759 0 0) 0x7f43e806f260 con 0x7f43e80b1f60 2024-03-29T11:25:47.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 >> v1:172.21.15.19:6790/0 conn(0x7f43e8143d10 legacy=0x7f43e8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 >> v1:172.21.15.62:6789/0 conn(0x7f43e8071eb0 legacy=0x7f43e8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:47.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 monclient: found mon.noname-a 2024-03-29T11:25:47.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:47.434 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:25:47.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8155d70 con 0x7f43e80b1f60 2024-03-29T11:25:47.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:47.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:47.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:47.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43efa27640 5 monclient: authenticate success, global_id 4458 2024-03-29T11:25:47.435 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43efa27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:47.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f43d8001670 con 0x7f43e80b1f60 2024-03-29T11:25:47.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:47.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:47.436 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: dump: 2024-03-29T11:25:47.436 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:47.436 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:47.437 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.437 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.437 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:47.437 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:47.437 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:47.437 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:47.438 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:47.438 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:47.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43efa27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:47.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43d8001890 con 0x7f43e80b1f60 2024-03-29T11:25:47.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.446+0000 7f43dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:47.438 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43efa27640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:47.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43efa27640 4 set_mon_vals no callback set 2024-03-29T11:25:47.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43dffff640 1 -- 172.21.15.19:0/139076843 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f43d8001c20 con 0x7f43e80b1f60 2024-03-29T11:25:47.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:47.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:47.439 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43dffff640 10 monclient: dump: 2024-03-29T11:25:47.439 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:47.440 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:47.441 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:47.441 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:47.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43efa27640 10 monclient: shutdown 2024-03-29T11:25:47.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43efa27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:47.441 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.447+0000 7f43efa27640 1 -- 172.21.15.19:0/139076843 >> v1:172.21.15.19:6789/0 conn(0x7f43e80b1f60 legacy=0x7f43e8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.451+0000 7f43efa27640 1 -- 172.21.15.19:0/139076843 shutdown_connections 2024-03-29T11:25:47.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.451+0000 7f43efa27640 1 -- 172.21.15.19:0/139076843 >> 172.21.15.19:0/139076843 conn(0x7f43e8066a00 msgr2=0x7f43e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:47.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.452+0000 7f43efa27640 1 -- 172.21.15.19:0/139076843 shutdown_connections 2024-03-29T11:25:47.452 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.452+0000 7f43efa27640 1 -- 172.21.15.19:0/139076843 wait complete. 2024-03-29T11:25:47.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.453+0000 7f43efa27640 10 monclient: build_initial_monmap 2024-03-29T11:25:47.453 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.453+0000 7f43efa27640 10 monclient: monmap: 2024-03-29T11:25:47.453 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:47.453 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:47.453 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:47.453 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:47.454 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:47.454 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:47.454 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:47.454 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:47.454 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:47.454 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:47.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.453+0000 7f43efa27640 1 librados: starting msgr at 2024-03-29T11:25:47.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.453+0000 7f43efa27640 1 librados: starting objecter 2024-03-29T11:25:47.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.454+0000 7f43efa27640 1 Processor -- start 2024-03-29T11:25:47.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.454+0000 7f43efa27640 1 -- start start 2024-03-29T11:25:47.455 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.454+0000 7f43efa27640 1 librados: setting wanted keys 2024-03-29T11:25:47.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.454+0000 7f43efa27640 1 librados: calling monclient init 2024-03-29T11:25:47.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.454+0000 7f43efa27640 10 monclient: init 2024-03-29T11:25:47.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:47.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:47.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient(hunting): picked mon.noname-b con 0x7f43e80b1f60 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:47.456 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient(hunting): picked mon.noname-c con 0x7f43e8143d10 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:47.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient(hunting): picked mon.noname-a con 0x7f43e8142c30 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:47.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e8142310 con 0x7f43e8142c30 2024-03-29T11:25:47.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e81537f0 con 0x7f43e80b1f60 2024-03-29T11:25:47.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e8153ad0 con 0x7f43e8143d10 2024-03-29T11:25:47.457 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:47.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.455+0000 7f43efa27640 10 monclient(hunting): authenticate will time out at 1267.718724s 2024-03-29T11:25:47.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ed7c3640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f43e80b1f60 0x7f43e8141aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:51600/0 (socket says 172.21.15.19:51600) 2024-03-29T11:25:47.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ed7c3640 1 -- 172.21.15.19:0/2119258074 learned_addr learned my addr 172.21.15.19:0/2119258074 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:47.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 client.?.objecter ms_handle_connect 0x7f43e80b1f60 2024-03-29T11:25:47.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:47.458 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 client.?.objecter ms_handle_connect 0x7f43e8143d10 2024-03-29T11:25:47.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:47.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 client.?.objecter ms_handle_connect 0x7f43e8142c30 2024-03-29T11:25:47.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:47.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568026554 0 0) 0x7f43e8142310 con 0x7f43e8142c30 2024-03-29T11:25:47.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:47.459 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:47.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:47.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:25:47.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.456+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc004eb0 con 0x7f43e8142c30 2024-03-29T11:25:47.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639106927 0 0) 0x7f43e8153ad0 con 0x7f43e8143d10 2024-03-29T11:25:47.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:47.460 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:47.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:47.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:25:47.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8142310 con 0x7f43e8143d10 2024-03-29T11:25:47.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790453090 0 0) 0x7f43e81537f0 con 0x7f43e80b1f60 2024-03-29T11:25:47.461 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:47.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:47.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:47.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:25:47.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8153ad0 con 0x7f43e80b1f60 2024-03-29T11:25:47.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2297656499 0 0) 0x7f43cc004eb0 con 0x7f43e8142c30 2024-03-29T11:25:47.462 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e81537f0 con 0x7f43e8142c30 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f43e0001a80 con 0x7f43e8142c30 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): dump: 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:47.463 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.464 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.464 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:47.464 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:47.464 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:47.464 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:47.464 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:47.465 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:47.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.457+0000 7f43ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:47.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969665201 0 0) 0x7f43e8153ad0 con 0x7f43e80b1f60 2024-03-29T11:25:47.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43cc004eb0 con 0x7f43e80b1f60 2024-03-29T11:25:47.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1923909435 0 0) 0x7f43e81537f0 con 0x7f43e8142c30 2024-03-29T11:25:47.465 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 >> v1:172.21.15.19:6790/0 conn(0x7f43e8143d10 legacy=0x7f43e8130b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 >> v1:172.21.15.62:6789/0 conn(0x7f43e80b1f60 legacy=0x7f43e8141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:47.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 monclient: found mon.a 2024-03-29T11:25:47.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:47.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:47.466 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8153db0 con 0x7f43e8142c30 2024-03-29T11:25:47.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:47.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:47.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:47.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:25:47.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 10 monclient: _renew_subs 2024-03-29T11:25:47.467 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:47.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e81efd30 con 0x7f43e8142c30 2024-03-29T11:25:47.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:47.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 10 monclient: _renew_subs 2024-03-29T11:25:47.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:47.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e81f00a0 con 0x7f43e8142c30 2024-03-29T11:25:47.468 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.458+0000 7f43efa27640 1 librados: init done 2024-03-29T11:25:47.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e0001be0 con 0x7f43e8142c30 2024-03-29T11:25:47.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:47.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f43e0003990 con 0x7f43e8142c30 2024-03-29T11:25:47.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:47.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43deffd640 4 set_mon_vals no callback set 2024-03-29T11:25:47.469 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:47.470 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.460+0000 7f43ddffb640 10 monclient: dump: 2024-03-29T11:25:47.470 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:47.470 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:47.470 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.470 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:47.470 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:47.471 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:47.471 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:47.471 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:47.471 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:47.471 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:47.471 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.461+0000 7f43efa27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.461+0000 7f43efa27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.461+0000 7f43efa27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.461+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c8005080 con 0x7f43e8142c30 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.462+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f43e0004be0 con 0x7f43e8142c30 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.463+0000 7f43ed7c3640 10 monclient: get_auth_request con 0x7f43cc06dd70 auth_method 0 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.463+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f43e0083940 con 0x7f43e8142c30 2024-03-29T11:25:47.472 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.463+0000 7f43ddffb640 10 client.4461.objecter ms_dispatch 0x7f43e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:47.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.463+0000 7f43ddffb640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:47.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.463+0000 7f43ddffb640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:47.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.463+0000 7f43ddffb640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:25:47.473 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.464+0000 7f43ddffb640 10 client.4461.objecter ms_handle_connect 0x7f43cc06dd70 2024-03-29T11:25:47.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.471+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43e004f050 con 0x7f43e8142c30 2024-03-29T11:25:47.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.471+0000 7f43ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:47.474 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.471+0000 7f43ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:47.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.895+0000 7f43efa27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:47.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.895+0000 7f43efa27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:47.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.895+0000 7f43efa27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:25:47.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.895+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43c8005330 con 0x7f43e8142c30 2024-03-29T11:25:47.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.898+0000 7f43ddffb640 1 -- 172.21.15.19:0/2119258074 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f43e0051150 con 0x7f43e8142c30 2024-03-29T11:25:47.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.898+0000 7f43ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:47.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.898+0000 7f43ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:47.883 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:47.883 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:47.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.902+0000 7f43efa27640 10 librados: watch_flush enter 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.902+0000 7f43efa27640 10 librados: watch_flush exit 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.902+0000 7f43efa27640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.902+0000 7f43efa27640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 >> v1:172.21.15.19:6816/3721232097 conn(0x7f43cc06dd70 legacy=0x7f43cc070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 10 monclient: shutdown 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:47.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 >> v1:172.21.15.19:6789/0 conn(0x7f43e8142c30 legacy=0x7f43e8131290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:47.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 shutdown_connections 2024-03-29T11:25:47.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 >> 172.21.15.19:0/2119258074 conn(0x7f43e8066a00 msgr2=0x7f43e8146980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:47.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 shutdown_connections 2024-03-29T11:25:47.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 -- 172.21.15.19:0/2119258074 wait complete. 2024-03-29T11:25:47.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:47.903+0000 7f43efa27640 1 librados: shutdown 2024-03-29T11:25:47.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:54.708 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:54.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.872+0000 7fe1e7280640 10 monclient: get_monmap_and_config 2024-03-29T11:25:54.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.872+0000 7fe1e7280640 10 monclient: build_initial_monmap 2024-03-29T11:25:54.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.872+0000 7fe1e7280640 10 monclient: monmap: 2024-03-29T11:25:54.857 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:54.857 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:54.857 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:25:54.873052+0000 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:25:54.873052+0000 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.873+0000 7fe1e7280640 1 Processor -- start 2024-03-29T11:25:54.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.873+0000 7fe1e7280640 1 -- start start 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.873+0000 7fe1e7280640 10 monclient: init 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d8006e30 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d8035e00 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d80c9670 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d8035970 con 0x7fe1d8035e00 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d8036280 con 0x7fe1d80c9670 2024-03-29T11:25:54.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d8035640 con 0x7fe1d8006e30 2024-03-29T11:25:54.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:54.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.874+0000 7fe1e7280640 10 monclient(hunting): authenticate will time out at 1275.137568s 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.878+0000 7fe1e627e640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe1d8006e30 0x7fe1d80cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:36025/0 (socket says 172.21.15.19:36025) 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.878+0000 7fe1e627e640 1 -- 172.21.15.19:0/224037261 learned_addr learned my addr 172.21.15.19:0/224037261 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.879+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562273482 0 0) 0x7fe1d8035640 con 0x7fe1d8006e30 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.879+0000 7fe1e527c640 10 monclient(hunting): _negotiate 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.879+0000 7fe1e527c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.879+0000 7fe1e527c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.879+0000 7fe1e527c640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:25:54.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.879+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1cc003520 con 0x7fe1d8006e30 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1593455028 0 0) 0x7fe1cc003520 con 0x7fe1d8006e30 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 >> v1:172.21.15.62:6789/0 conn(0x7fe1d80c9670 legacy=0x7fe1d80d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 >> v1:172.21.15.19:6789/0 conn(0x7fe1d8035e00 legacy=0x7fe1d80c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: _finish_hunting 0 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 monclient: found mon.noname-b 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d8034e20 con 0x7fe1d8006e30 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: _finish_auth 0 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: _check_auth_tickets 2024-03-29T11:25:54.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe1e004f730 con 0x7fe1d8006e30 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: dump: 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:54.866 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1e0075f10 con 0x7fe1d8006e30 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 1 -- 172.21.15.19:0/224037261 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe1e004f730 con 0x7fe1d8006e30 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:54.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.880+0000 7fe1e527c640 10 monclient: dump: 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.881+0000 7fe1e7280640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.881+0000 7fe1e7280640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:54.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.881+0000 7fe1e7280640 4 set_mon_vals no callback set 2024-03-29T11:25:54.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.882+0000 7fe1e7280640 10 monclient: shutdown 2024-03-29T11:25:54.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.882+0000 7fe1e7280640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:54.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.882+0000 7fe1e7280640 1 -- 172.21.15.19:0/224037261 >> v1:172.21.15.19:6790/0 conn(0x7fe1d8006e30 legacy=0x7fe1d80cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:54.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.883+0000 7fe1e7280640 1 -- 172.21.15.19:0/224037261 shutdown_connections 2024-03-29T11:25:54.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.883+0000 7fe1e7280640 1 -- 172.21.15.19:0/224037261 >> 172.21.15.19:0/224037261 conn(0x7fe1d8005c00 msgr2=0x7fe1d8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:54.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.883+0000 7fe1e7280640 1 -- 172.21.15.19:0/224037261 shutdown_connections 2024-03-29T11:25:54.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.883+0000 7fe1e7280640 1 -- 172.21.15.19:0/224037261 wait complete. 2024-03-29T11:25:54.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.884+0000 7fe1e7280640 10 monclient: build_initial_monmap 2024-03-29T11:25:54.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.884+0000 7fe1e7280640 10 monclient: monmap: 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.884+0000 7fe1e7280640 1 librados: starting msgr at 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.884+0000 7fe1e7280640 1 librados: starting objecter 2024-03-29T11:25:54.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.884+0000 7fe1e7280640 1 Processor -- start 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 1 -- start start 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 1 librados: setting wanted keys 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 1 librados: calling monclient init 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient: init 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient(hunting): picked mon.noname-c con 0x7fe1d8006e30 addr v1:172.21.15.19:6790/0 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient(hunting): picked mon.noname-a con 0x7fe1d8035e00 addr v1:172.21.15.19:6789/0 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient(hunting): picked mon.noname-b con 0x7fe1d80c9670 addr v1:172.21.15.62:6789/0 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80c0820 con 0x7fe1d8035e00 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d8030130 con 0x7fe1d80c9670 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d802d150 con 0x7fe1d8006e30 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:54.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.885+0000 7fe1e7280640 10 monclient(hunting): authenticate will time out at 1275.148844s 2024-03-29T11:25:54.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.893+0000 7fe1e6a7f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe1d80c9670 0x7fe1d80bce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:34729/0 (socket says 172.21.15.19:34729) 2024-03-29T11:25:54.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.893+0000 7fe1e6a7f640 1 -- 172.21.15.19:0/2155152628 learned_addr learned my addr 172.21.15.19:0/2155152628 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:25:54.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.894+0000 7fe1d6ffd640 10 client.?.objecter ms_handle_connect 0x7fe1d8006e30 2024-03-29T11:25:54.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.894+0000 7fe1d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:54.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.894+0000 7fe1d6ffd640 10 client.?.objecter ms_handle_connect 0x7fe1d8035e00 2024-03-29T11:25:54.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.894+0000 7fe1d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:54.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601689638 0 0) 0x7fe1d80c0820 con 0x7fe1d8035e00 2024-03-29T11:25:54.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:54.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:54.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:54.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:25:54.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4004eb0 con 0x7fe1d8035e00 2024-03-29T11:25:54.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482727273 0 0) 0x7fe1d802d150 con 0x7fe1d8006e30 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1d80c0820 con 0x7fe1d8006e30 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 client.?.objecter ms_handle_connect 0x7fe1d80c9670 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.895+0000 7fe1d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.898+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903232905 0 0) 0x7fe1d80c0820 con 0x7fe1d8006e30 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.898+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d802d150 con 0x7fe1d8006e30 2024-03-29T11:25:54.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.898+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2792771983 0 0) 0x7fe1c4004eb0 con 0x7fe1d8035e00 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.898+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1d80c0820 con 0x7fe1d8035e00 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230028037 0 0) 0x7fe1d8030130 con 0x7fe1d80c9670 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4004eb0 con 0x7fe1d80c9670 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe1e004f730 con 0x7fe1d8006e30 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:54.885 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe1d00017f0 con 0x7fe1d8035e00 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:54.886 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:54.887 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2932446725 0 0) 0x7fe1d802d150 con 0x7fe1d8006e30 2024-03-29T11:25:54.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 >> v1:172.21.15.62:6789/0 conn(0x7fe1d80c9670 legacy=0x7fe1d80bce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:54.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 >> v1:172.21.15.19:6789/0 conn(0x7fe1d8035e00 legacy=0x7fe1d80bc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 monclient: found mon.c 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d802d2e0 con 0x7fe1d8006e30 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:54.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.899+0000 7fe1d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:54.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1e0068810 con 0x7fe1d8006e30 2024-03-29T11:25:54.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:54.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:25:54.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 10 monclient: _renew_subs 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d80c0190 con 0x7fe1d8006e30 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 10 monclient: _renew_subs 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d8186660 con 0x7fe1d8006e30 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1e7280640 1 librados: init done 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.900+0000 7fe1d7fff640 4 set_mon_vals no callback set 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.901+0000 7fe1e7280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1e7280640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:54.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1e7280640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d8007b90 con 0x7fe1d8006e30 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe1e0067bf0 con 0x7fe1d8006e30 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.902+0000 7fe1d6ffd640 10 monclient: dump: 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:25:54.893 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:25:54.894 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:25:54.894 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:25:54.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.907+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fe1e0067d50 con 0x7fe1d8006e30 2024-03-29T11:25:54.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fe1e00ff1a0 con 0x7fe1d8006e30 2024-03-29T11:25:54.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1d6ffd640 10 client.4481.objecter ms_dispatch 0x7fe1e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:54.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1d6ffd640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:54.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1d6ffd640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:54.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1d6ffd640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:25:54.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1e5a7d640 10 monclient: get_auth_request con 0x7fe1c406e040 auth_method 0 2024-03-29T11:25:54.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.908+0000 7fe1d6ffd640 10 client.4481.objecter ms_handle_connect 0x7fe1c406e040 2024-03-29T11:25:54.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.913+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1e00cab30 con 0x7fe1d8006e30 2024-03-29T11:25:54.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.913+0000 7fe1d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:54.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:54.913+0000 7fe1d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:55.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.233+0000 7fe1e7280640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:55.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.233+0000 7fe1e7280640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:55.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.233+0000 7fe1e7280640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:25:55.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.233+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1d8007b90 con 0x7fe1d8006e30 2024-03-29T11:25:55.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.234+0000 7fe1d6ffd640 1 -- 172.21.15.19:0/2155152628 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fe1e00cca60 con 0x7fe1d8006e30 2024-03-29T11:25:55.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.234+0000 7fe1d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:55.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.234+0000 7fe1d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:55.220 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:25:55.220 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:55.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 10 librados: watch_flush enter 2024-03-29T11:25:55.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 10 librados: watch_flush exit 2024-03-29T11:25:55.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:25:55.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:25:55.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 >> v1:172.21.15.19:6816/3721232097 conn(0x7fe1c406e040 legacy=0x7fe1c4070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:55.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 10 monclient: shutdown 2024-03-29T11:25:55.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:55.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.238+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 >> v1:172.21.15.19:6790/0 conn(0x7fe1d8006e30 legacy=0x7fe1d80cb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:55.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.239+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 shutdown_connections 2024-03-29T11:25:55.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.239+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 >> 172.21.15.19:0/2155152628 conn(0x7fe1d8005c00 msgr2=0x7fe1d80c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:55.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.239+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 shutdown_connections 2024-03-29T11:25:55.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.242+0000 7fe1e7280640 1 -- 172.21.15.19:0/2155152628 wait complete. 2024-03-29T11:25:55.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:25:55.242+0000 7fe1e7280640 1 librados: shutdown 2024-03-29T11:25:55.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:02.051 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:02.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.217+0000 7f79200e6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:02.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.217+0000 7f79200e6640 10 monclient: build_initial_monmap 2024-03-29T11:26:02.202 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.217+0000 7f79200e6640 10 monclient: monmap: 2024-03-29T11:26:02.202 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:02.218178+0000 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:02.218178+0000 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:02.203 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.218+0000 7f79200e6640 1 Processor -- start 2024-03-29T11:26:02.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.218+0000 7f79200e6640 1 -- start start 2024-03-29T11:26:02.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.218+0000 7f79200e6640 10 monclient: init 2024-03-29T11:26:02.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f79200e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:02.204 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f79200e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f79200e6640 10 monclient(hunting): picked mon.noname-c con 0x7f791806f050 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f791de82640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f791806f050 0x7f79181422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:42812/0 (socket says 172.21.15.19:42812) 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f791de82640 1 -- 172.21.15.19:0/832180624 learned_addr learned my addr 172.21.15.19:0/832180624 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f79200e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7918070240 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.219+0000 7f79200e6640 10 monclient(hunting): picked mon.noname-b con 0x7f791806f430 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791813f400 con 0x7f7918070240 2024-03-29T11:26:02.206 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79181429c0 con 0x7f791806f050 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791806ff80 con 0x7f791806f430 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 10 monclient(hunting): authenticate will time out at 1282.483194s 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927834569 0 0) 0x7f791813f400 con 0x7f7918070240 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient(hunting): _negotiate 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7904003520 con 0x7f7918070240 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1797769289 0 0) 0x7f7904003520 con 0x7f7918070240 2024-03-29T11:26:02.207 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 >> v1:172.21.15.19:6790/0 conn(0x7f791806f430 legacy=0x7f791814a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 >> v1:172.21.15.62:6789/0 conn(0x7f791806f050 legacy=0x7f79181422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: _finish_hunting 0 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 monclient: found mon.noname-a 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180aef70 con 0x7f7918070240 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: _finish_auth 0 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: _check_auth_tickets 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:02.208 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7908001690 con 0x7f7918070240 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f791ce80640 10 monclient: dump: 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.209 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.220+0000 7f79200e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7908001860 con 0x7f7918070240 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f791ce80640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f791ce80640 1 -- 172.21.15.19:0/832180624 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7908001c70 con 0x7f7918070240 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f791ce80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:02.210 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f791ce80640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f791ce80640 10 monclient: dump: 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f79200e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:02.211 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.221+0000 7f79200e6640 4 set_mon_vals no callback set 2024-03-29T11:26:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 10 monclient: shutdown 2024-03-29T11:26:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 >> v1:172.21.15.19:6789/0 conn(0x7f7918070240 legacy=0x7f79180ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 shutdown_connections 2024-03-29T11:26:02.217 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 >> 172.21.15.19:0/832180624 conn(0x7f79180b4930 msgr2=0x7f79180b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 shutdown_connections 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.222+0000 7f79200e6640 1 -- 172.21.15.19:0/832180624 wait complete. 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.223+0000 7f79200e6640 10 monclient: build_initial_monmap 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.223+0000 7f79200e6640 10 monclient: monmap: 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:02.218 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 1 librados: starting msgr at 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 1 librados: starting objecter 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 1 Processor -- start 2024-03-29T11:26:02.219 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 1 -- start start 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 1 librados: setting wanted keys 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 1 librados: calling monclient init 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.224+0000 7f79200e6640 10 monclient: init 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 10 monclient(hunting): picked mon.noname-c con 0x7f791806f050 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 10 monclient(hunting): picked mon.noname-a con 0x7f791806f430 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:02.220 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 10 monclient(hunting): picked mon.noname-b con 0x7f7918070240 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918138ad0 con 0x7f791806f430 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f79200e6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79181411c0 con 0x7f7918070240 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.225+0000 7f791de82640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f791806f050 0x7f79181d71f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52580/0 (socket says 172.21.15.19:52580) 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f791de82640 1 -- 172.21.15.19:0/2026170782 learned_addr learned my addr 172.21.15.19:0/2026170782 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 client.?.objecter ms_handle_connect 0x7f791806f050 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79181422c0 con 0x7f791806f050 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f79200e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 client.?.objecter ms_handle_connect 0x7f791806f430 2024-03-29T11:26:02.221 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 client.?.objecter ms_handle_connect 0x7f7918070240 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f79200e6640 10 monclient(hunting): authenticate will time out at 1282.489300s 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187991964 0 0) 0x7f79181411c0 con 0x7f7918070240 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc004eb0 con 0x7f7918070240 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189237473 0 0) 0x7f79181422c0 con 0x7f791806f050 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:02.222 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79181411c0 con 0x7f791806f050 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280203305 0 0) 0x7f7918138ad0 con 0x7f791806f430 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.226+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79181422c0 con 0x7f791806f430 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1243287289 0 0) 0x7f78fc004eb0 con 0x7f7918070240 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7918138ad0 con 0x7f7918070240 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7910001b70 con 0x7f7918070240 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:02.223 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3835229267 0 0) 0x7f79181422c0 con 0x7f791806f430 2024-03-29T11:26:02.224 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78fc004eb0 con 0x7f791806f430 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7908001b70 con 0x7f791806f430 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:02.225 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3511566296 0 0) 0x7f79181411c0 con 0x7f791806f050 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79181422c0 con 0x7f791806f050 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f79140018f0 con 0x7f791806f050 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.227+0000 7f790e7fc640 10 monclient(hunting): dump: 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.226 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3787690708 0 0) 0x7f7918138ad0 con 0x7f7918070240 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 >> v1:172.21.15.19:6790/0 conn(0x7f791806f050 legacy=0x7f79181d71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 >> v1:172.21.15.19:6789/0 conn(0x7f791806f430 legacy=0x7f791812fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 1 monclient: found mon.b 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:02.227 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918141950 con 0x7f7918070240 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f790e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 10 monclient: _renew_subs 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7918137890 con 0x7f7918070240 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 10 monclient: _renew_subs 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79181d7ae0 con 0x7f7918070240 2024-03-29T11:26:02.228 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.228+0000 7f79200e6640 1 librados: init done 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7910002080 con 0x7f7918070240 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7910003b10 con 0x7f7918070240 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790e7fc640 10 monclient: dump: 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:02.229 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.229+0000 7f790f7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.230+0000 7f79200e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.230+0000 7f79200e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.230+0000 7f79200e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:02.230 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.230+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7900005080 con 0x7f7918070240 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.231+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f7910019b60 con 0x7f7918070240 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.231+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f7910083b00 con 0x7f7918070240 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.231+0000 7f790e7fc640 10 client.4456.objecter ms_dispatch 0x7f7918000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.231+0000 7f790e7fc640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.231+0000 7f790e7fc640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.232+0000 7f790e7fc640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.232+0000 7f791de82640 10 monclient: get_auth_request con 0x7f78fc072500 auth_method 0 2024-03-29T11:26:02.231 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.232+0000 7f790e7fc640 10 client.4456.objecter ms_handle_connect 0x7f78fc072500 2024-03-29T11:26:02.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.239+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f791004f210 con 0x7f7918070240 2024-03-29T11:26:02.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.239+0000 7f790e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:02.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.239+0000 7f790e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:02.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.554+0000 7f79200e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:02.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.554+0000 7f79200e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:02.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.554+0000 7f79200e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:02.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.554+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7900005330 con 0x7f7918070240 2024-03-29T11:26:02.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.555+0000 7f790e7fc640 1 -- 172.21.15.19:0/2026170782 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f7910051310 con 0x7f7918070240 2024-03-29T11:26:02.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.555+0000 7f790e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:02.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.555+0000 7f790e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:02.541 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:02.541 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 10 librados: watch_flush enter 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 10 librados: watch_flush exit 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 10 client.4456.objecter successfully canceled tick 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 >> v1:172.21.15.19:6816/3721232097 conn(0x7f78fc072500 legacy=0x7f78fc0749c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 10 monclient: shutdown 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 >> v1:172.21.15.62:6789/0 conn(0x7f7918070240 legacy=0x7f79181df530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 shutdown_connections 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 >> 172.21.15.19:0/2026170782 conn(0x7f79180b4930 msgr2=0x7f7918072040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:02.545 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 shutdown_connections 2024-03-29T11:26:02.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 -- 172.21.15.19:0/2026170782 wait complete. 2024-03-29T11:26:02.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:02.559+0000 7f79200e6640 1 librados: shutdown 2024-03-29T11:26:02.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:09.366 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.511+0000 7fd70cd3d640 10 monclient: get_monmap_and_config 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.511+0000 7fd70cd3d640 10 monclient: build_initial_monmap 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.511+0000 7fd70cd3d640 10 monclient: monmap: 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:09.512344+0000 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:09.512344+0000 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:09.497 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.498 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:09.498 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:09.498 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:09.498 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.512+0000 7fd70cd3d640 1 Processor -- start 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.512+0000 7fd70cd3d640 1 -- start start 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.512+0000 7fd70cd3d640 10 monclient: init 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7081422b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient(hunting): picked mon.noname-b con 0x7fd708142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:09.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient(hunting): picked mon.noname-a con 0x7fd7080b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd708142c80 con 0x7fd7080b17d0 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7080aae40 con 0x7fd7081422b0 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70813fbd0 con 0x7fd708142690 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd70cd3d640 10 monclient(hunting): authenticate will time out at 1289.776772s 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd705d9b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd708142690 0x7fd7080b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33512/0 (socket says 172.21.15.19:33512) 2024-03-29T11:26:09.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.513+0000 7fd705d9b640 1 -- 172.21.15.19:0/200808224 learned_addr learned my addr 172.21.15.19:0/200808224 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569781311 0 0) 0x7fd70813fbd0 con 0x7fd708142690 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6f0003520 con 0x7fd708142690 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085552988 0 0) 0x7fd7080aae40 con 0x7fd7081422b0 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:09.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:26:09.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd70813fbd0 con 0x7fd7081422b0 2024-03-29T11:26:09.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862552176 0 0) 0x7fd6f0003520 con 0x7fd708142690 2024-03-29T11:26:09.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 >> v1:172.21.15.62:6789/0 conn(0x7fd7081422b0 legacy=0x7fd70813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:09.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.514+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 >> v1:172.21.15.19:6789/0 conn(0x7fd7080b17d0 legacy=0x7fd7080b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:09.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:09.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 1 monclient: found mon.noname-b 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd708151940 con 0x7fd708142690 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: _finish_auth 0 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70cd3d640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70cd3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd6f8001a10 con 0x7fd708142690 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70559a640 10 monclient: dump: 2024-03-29T11:26:09.505 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.517+0000 7fd70cd3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6f8001b70 con 0x7fd708142690 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:09.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70559a640 1 -- 172.21.15.19:0/200808224 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd6f8001f00 con 0x7fd708142690 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70cd3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70cd3d640 4 set_mon_vals no callback set 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70559a640 10 monclient: dump: 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.507 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70cd3d640 10 monclient: shutdown 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70cd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.518+0000 7fd70cd3d640 1 -- 172.21.15.19:0/200808224 >> v1:172.21.15.19:6790/0 conn(0x7fd708142690 legacy=0x7fd7080b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.519+0000 7fd70cd3d640 1 -- 172.21.15.19:0/200808224 shutdown_connections 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.519+0000 7fd70cd3d640 1 -- 172.21.15.19:0/200808224 >> 172.21.15.19:0/200808224 conn(0x7fd708066a00 msgr2=0x7fd708066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.519+0000 7fd70cd3d640 1 -- 172.21.15.19:0/200808224 shutdown_connections 2024-03-29T11:26:09.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.519+0000 7fd70cd3d640 1 -- 172.21.15.19:0/200808224 wait complete. 2024-03-29T11:26:09.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.520+0000 7fd70cd3d640 10 monclient: build_initial_monmap 2024-03-29T11:26:09.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.520+0000 7fd70cd3d640 10 monclient: monmap: 2024-03-29T11:26:09.516 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:09.516 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 1 librados: starting msgr at 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 1 librados: starting objecter 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 1 Processor -- start 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 1 -- start start 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 1 librados: setting wanted keys 2024-03-29T11:26:09.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 1 librados: calling monclient init 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.521+0000 7fd70cd3d640 10 monclient: init 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient(hunting): picked mon.noname-c con 0x7fd7080b17d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient(hunting): picked mon.noname-b con 0x7fd7081422b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient(hunting): picked mon.noname-a con 0x7fd708142690 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7080acaf0 con 0x7fd708142690 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7080ad040 con 0x7fd7081422b0 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7080acd60 con 0x7fd7080b17d0 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70cd3d640 10 monclient(hunting): authenticate will time out at 1289.785383s 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70659c640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fd7080b17d0 0x7fd708129d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:35488/0 (socket says 172.21.15.19:35488) 2024-03-29T11:26:09.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.522+0000 7fd70659c640 1 -- 172.21.15.19:0/4198800669 learned_addr learned my addr 172.21.15.19:0/4198800669 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd7080b17d0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215252099 0 0) 0x7fd7080acd60 con 0x7fd7080b17d0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.523+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e8004eb0 con 0x7fd7080b17d0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 client.?.objecter ms_handle_connect 0x7fd7081422b0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1408586466 0 0) 0x7fd6e8004eb0 con 0x7fd7080b17d0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd6e8005300 con 0x7fd7080b17d0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982945781 0 0) 0x7fd7080ad040 con 0x7fd7081422b0 2024-03-29T11:26:09.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6e8005760 con 0x7fd7081422b0 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd7000019f0 con 0x7fd7080b17d0 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.520 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3380576471 0 0) 0x7fd6e8005760 con 0x7fd7081422b0 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7080ad040 con 0x7fd7081422b0 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd6f8001e80 con 0x7fd7081422b0 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:09.521 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3248121334 0 0) 0x7fd6e8005300 con 0x7fd7080b17d0 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 >> v1:172.21.15.62:6789/0 conn(0x7fd7081422b0 legacy=0x7fd7081daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.524+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 >> v1:172.21.15.19:6789/0 conn(0x7fd708142690 legacy=0x7fd7081db180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:09.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 1 monclient: found mon.c 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7080abd30 con 0x7fd7080b17d0 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd6f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 10 monclient: _renew_subs 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7080aba00 con 0x7fd7080b17d0 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 10 monclient: _renew_subs 2024-03-29T11:26:09.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd708132320 con 0x7fd7080b17d0 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.525+0000 7fd70cd3d640 1 librados: init done 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.526+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd700001cb0 con 0x7fd7080b17d0 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.526+0000 7fd6f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.526+0000 7fd704d99640 4 set_mon_vals no callback set 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.527+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fd7000019f0 con 0x7fd7080b17d0 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.527+0000 7fd6f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.527+0000 7fd6f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.527+0000 7fd6f6ffd640 10 monclient: dump: 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:09.524 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.528+0000 7fd70cd3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.528+0000 7fd70cd3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.528+0000 7fd70cd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.528+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ec005080 con 0x7fd7080b17d0 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.528+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fd7000038c0 con 0x7fd7080b17d0 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.529+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fd7000845e0 con 0x7fd7080b17d0 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.529+0000 7fd6f6ffd640 10 client.4490.objecter ms_dispatch 0x7fd708000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.529+0000 7fd6f6ffd640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:09.525 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.529+0000 7fd6f6ffd640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:09.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.529+0000 7fd6f6ffd640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:26:09.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.529+0000 7fd705d9b640 10 monclient: get_auth_request con 0x7fd6e806e5f0 auth_method 0 2024-03-29T11:26:09.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.530+0000 7fd6f6ffd640 10 client.4490.objecter ms_handle_connect 0x7fd6e806e5f0 2024-03-29T11:26:09.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.536+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd70004ff20 con 0x7fd7080b17d0 2024-03-29T11:26:09.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.536+0000 7fd6f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:09.526 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.536+0000 7fd6f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.855+0000 7fd70cd3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.855+0000 7fd70cd3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.855+0000 7fd70cd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:09.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.855+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6ec005330 con 0x7fd7080b17d0 2024-03-29T11:26:09.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.856+0000 7fd6f6ffd640 1 -- 172.21.15.19:0/4198800669 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fd700001850 con 0x7fd7080b17d0 2024-03-29T11:26:09.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.856+0000 7fd6f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:09.846 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.856+0000 7fd6f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:09.846 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:09.846 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.859+0000 7fd70cd3d640 10 librados: watch_flush enter 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.859+0000 7fd70cd3d640 10 librados: watch_flush exit 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.859+0000 7fd70cd3d640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 >> v1:172.21.15.19:6816/3721232097 conn(0x7fd6e806e5f0 legacy=0x7fd6e8070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 10 monclient: shutdown 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 >> v1:172.21.15.19:6790/0 conn(0x7fd7080b17d0 legacy=0x7fd708129d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:09.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 shutdown_connections 2024-03-29T11:26:09.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 >> 172.21.15.19:0/4198800669 conn(0x7fd708066a00 msgr2=0x7fd70813c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:09.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 shutdown_connections 2024-03-29T11:26:09.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 -- 172.21.15.19:0/4198800669 wait complete. 2024-03-29T11:26:09.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:09.860+0000 7fd70cd3d640 1 librados: shutdown 2024-03-29T11:26:09.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:16.669 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.817+0000 7f507f3d6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.817+0000 7f507f3d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:16.804 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.817+0000 7f507f3d6640 10 monclient: monmap: 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:16.819294+0000 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:16.819294+0000 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:16.805 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:16.806 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:16.806 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:16.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.818+0000 7f507f3d6640 1 Processor -- start 2024-03-29T11:26:16.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.818+0000 7f507f3d6640 1 -- start start 2024-03-29T11:26:16.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.818+0000 7f507f3d6640 10 monclient: init 2024-03-29T11:26:16.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.819+0000 7f507f3d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:16.809 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.819+0000 7f507f3d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 10 monclient(hunting): picked mon.noname-a con 0x7f50781457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 10 monclient(hunting): picked mon.noname-c con 0x7f507813f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 10 monclient(hunting): picked mon.noname-b con 0x7f50780a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50780aaef0 con 0x7f50781457f0 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5078145bd0 con 0x7f507813f650 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507d973640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f50780a9d70 0x7f507814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:49466/0 (socket says 172.21.15.19:49466) 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507d973640 1 -- 172.21.15.19:0/3197750910 learned_addr learned my addr 172.21.15.19:0/3197750910 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 1 -- 172.21.15.19:0/3197750910 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5078145eb0 con 0x7f50780a9d70 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:16.810 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.820+0000 7f507f3d6640 10 monclient(hunting): authenticate will time out at 1297.084827s 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023861436 0 0) 0x7f5078145eb0 con 0x7f50780a9d70 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5064003520 con 0x7f50780a9d70 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 767618291 0 0) 0x7f5064003520 con 0x7f50780a9d70 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 >> v1:172.21.15.62:6789/0 conn(0x7f507813f650 legacy=0x7f50780b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.821+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 >> v1:172.21.15.19:6789/0 conn(0x7f50781457f0 legacy=0x7f507813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:16.811 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 1 monclient: found mon.noname-b 2024-03-29T11:26:16.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:16.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:26:16.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5078146340 con 0x7f50780a9d70 2024-03-29T11:26:16.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:16.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:16.812 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5070001a10 con 0x7f50780a9d70 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f506ffff640 10 monclient: dump: 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:16.813 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f507f3d6640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.822+0000 7f507f3d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5070001be0 con 0x7f50780a9d70 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f506ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f507f3d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:16.814 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f507f3d6640 4 set_mon_vals no callback set 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f506ffff640 1 -- 172.21.15.19:0/3197750910 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5070002030 con 0x7f50780a9d70 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f506ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f506ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.823+0000 7f506ffff640 10 monclient: dump: 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:16.821 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 10 monclient: shutdown 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 1 -- 172.21.15.19:0/3197750910 >> v1:172.21.15.19:6790/0 conn(0x7f50780a9d70 legacy=0x7f507814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 1 -- 172.21.15.19:0/3197750910 shutdown_connections 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 1 -- 172.21.15.19:0/3197750910 >> 172.21.15.19:0/3197750910 conn(0x7f5078066a00 msgr2=0x7f5078066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 1 -- 172.21.15.19:0/3197750910 shutdown_connections 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.824+0000 7f507f3d6640 1 -- 172.21.15.19:0/3197750910 wait complete. 2024-03-29T11:26:16.822 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 10 monclient: monmap: 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:16.823 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 1 librados: starting msgr at 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 1 librados: starting objecter 2024-03-29T11:26:16.824 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 1 Processor -- start 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 1 -- start start 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 1 librados: setting wanted keys 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 1 librados: calling monclient init 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.825+0000 7f507f3d6640 10 monclient: init 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient(hunting): picked mon.noname-a con 0x7f50780a9d70 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:16.825 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient(hunting): picked mon.noname-b con 0x7f507813f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:16.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient(hunting): picked mon.noname-c con 0x7f50781457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:16.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5078142150 con 0x7f50780a9d70 2024-03-29T11:26:16.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507d973640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f50781457f0 0x7f5078137210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52650/0 (socket says 172.21.15.19:52650) 2024-03-29T11:26:16.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507d973640 1 -- 172.21.15.19:0/726605735 learned_addr learned my addr 172.21.15.19:0/726605735 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:16.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50780ac780 con 0x7f507813f650 2024-03-29T11:26:16.826 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5078137920 con 0x7f50781457f0 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f506dffb640 10 client.?.objecter ms_handle_connect 0x7f50781457f0 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f506dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.826+0000 7f507f3d6640 10 monclient(hunting): authenticate will time out at 1297.090813s 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475071554 0 0) 0x7f5078137920 con 0x7f50781457f0 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:16.827 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:16.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:16.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:26:16.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f505c004eb0 con 0x7f50781457f0 2024-03-29T11:26:16.828 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1325418767 0 0) 0x7f505c004eb0 con 0x7f50781457f0 2024-03-29T11:26:16.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5078137920 con 0x7f50781457f0 2024-03-29T11:26:16.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5070001ef0 con 0x7f50781457f0 2024-03-29T11:26:16.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:16.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:16.829 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:16.829 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:16.830 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 547177326 0 0) 0x7f5078137920 con 0x7f50781457f0 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 >> v1:172.21.15.62:6789/0 conn(0x7f507813f650 legacy=0x7f5078136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.827+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 >> v1:172.21.15.19:6789/0 conn(0x7f50780a9d70 legacy=0x7f507813fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:16.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 1 monclient: found mon.c 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50780ac9f0 con 0x7f50781457f0 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:26:16.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 10 monclient: _renew_subs 2024-03-29T11:26:16.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:16.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5078141dc0 con 0x7f50781457f0 2024-03-29T11:26:16.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5070004f70 con 0x7f50781457f0 2024-03-29T11:26:16.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:16.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5070005280 con 0x7f50781457f0 2024-03-29T11:26:16.833 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506dffb640 10 monclient: dump: 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:16.834 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:16.835 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:16.835 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:16.835 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:16.835 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 10 monclient: _renew_subs 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f506f7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5078142bf0 con 0x7f50781457f0 2024-03-29T11:26:16.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.830+0000 7f507f3d6640 1 librados: init done 2024-03-29T11:26:16.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f507001b250 con 0x7f50781457f0 2024-03-29T11:26:16.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f507f3d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:16.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f507f3d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:16.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f507f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:16.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5078142910 con 0x7f50781457f0 2024-03-29T11:26:16.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f5070085010 con 0x7f50781457f0 2024-03-29T11:26:16.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f506dffb640 10 client.4496.objecter ms_dispatch 0x7f5078000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:16.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f506dffb640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:16.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f506dffb640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:16.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f507d172640 10 monclient: get_auth_request con 0x7f505c06db30 auth_method 0 2024-03-29T11:26:16.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.833+0000 7f506dffb640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:26:16.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.834+0000 7f506dffb640 10 client.4496.objecter ms_handle_connect 0x7f505c06db30 2024-03-29T11:26:16.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.841+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50700508f0 con 0x7f50781457f0 2024-03-29T11:26:16.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.841+0000 7f506dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:16.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:16.841+0000 7f506dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:17.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.175+0000 7f507f3d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:17.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.175+0000 7f507f3d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:17.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.175+0000 7f507f3d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:17.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.175+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5078075f10 con 0x7f50781457f0 2024-03-29T11:26:17.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.176+0000 7f506dffb640 1 -- 172.21.15.19:0/726605735 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f5070052820 con 0x7f50781457f0 2024-03-29T11:26:17.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.176+0000 7f506dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:17.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.176+0000 7f506dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:17.164 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:17.164 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:17.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.180+0000 7f507f3d6640 10 librados: watch_flush enter 2024-03-29T11:26:17.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.180+0000 7f507f3d6640 10 librados: watch_flush exit 2024-03-29T11:26:17.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.180+0000 7f507f3d6640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:26:17.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.180+0000 7f507f3d6640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:26:17.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 >> v1:172.21.15.19:6816/3721232097 conn(0x7f505c06db30 legacy=0x7f505c06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:17.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 10 monclient: shutdown 2024-03-29T11:26:17.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:17.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 >> v1:172.21.15.19:6790/0 conn(0x7f50781457f0 legacy=0x7f5078137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:17.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 shutdown_connections 2024-03-29T11:26:17.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 >> 172.21.15.19:0/726605735 conn(0x7f5078066a00 msgr2=0x7f507813abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:17.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 shutdown_connections 2024-03-29T11:26:17.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 -- 172.21.15.19:0/726605735 wait complete. 2024-03-29T11:26:17.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:17.183+0000 7f507f3d6640 1 librados: shutdown 2024-03-29T11:26:17.187 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:23.994 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:24.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.165+0000 7ff39e2d9640 10 monclient: get_monmap_and_config 2024-03-29T11:26:24.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.165+0000 7ff39e2d9640 10 monclient: build_initial_monmap 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.165+0000 7ff39e2d9640 10 monclient: monmap: 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:24.167460+0000 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:24.167460+0000 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:24.153 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.166+0000 7ff39e2d9640 1 Processor -- start 2024-03-29T11:26:24.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 1 -- start start 2024-03-29T11:26:24.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient: init 2024-03-29T11:26:24.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:24.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff3900389e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff390038dc0 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff390030660 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3900c54c0 con 0x7ff390030660 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff390036430 con 0x7ff3900389e0 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff390039840 con 0x7ff390038dc0 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.167+0000 7ff39e2d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:24.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.168+0000 7ff39e2d9640 10 monclient(hunting): authenticate will time out at 1304.432002s 2024-03-29T11:26:24.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.171+0000 7ff39dad8640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff390030660 0x7ff3900d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:45581/0 (socket says 172.21.15.19:45581) 2024-03-29T11:26:24.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.171+0000 7ff39dad8640 1 -- 172.21.15.19:0/1351299395 learned_addr learned my addr 172.21.15.19:0/1351299395 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:24.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.172+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775656812 0 0) 0x7ff3900c54c0 con 0x7ff390030660 2024-03-29T11:26:24.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.172+0000 7ff38ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.172+0000 7ff38ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.172+0000 7ff38ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.172+0000 7ff38ffff640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.172+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff384003520 con 0x7ff390030660 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.173+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2601587274 0 0) 0x7ff384003520 con 0x7ff390030660 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.173+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 >> v1:172.21.15.19:6790/0 conn(0x7ff390038dc0 legacy=0x7ff3900cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:24.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.173+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 >> v1:172.21.15.62:6789/0 conn(0x7ff3900389e0 legacy=0x7ff390033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:24.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 1 monclient: found mon.noname-a 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390035690 con 0x7ff390030660 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff39804f730 con 0x7ff390030660 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.175+0000 7ff38ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:24.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.176+0000 7ff38ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:24.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.176+0000 7ff38ffff640 10 monclient: dump: 2024-03-29T11:26:24.165 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:24.165 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:24.165 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.176+0000 7ff39e2d9640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:26:24.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.176+0000 7ff39e2d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:24.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff398075f10 con 0x7ff390030660 2024-03-29T11:26:24.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff38ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:24.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff38ffff640 1 -- 172.21.15.19:0/1351299395 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff3980758b0 con 0x7ff390030660 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff38ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff38ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff38ffff640 10 monclient: dump: 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff39e2d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:24.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.179+0000 7ff39e2d9640 4 set_mon_vals no callback set 2024-03-29T11:26:24.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.180+0000 7ff39e2d9640 10 monclient: shutdown 2024-03-29T11:26:24.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.180+0000 7ff39e2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:24.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.180+0000 7ff39e2d9640 1 -- 172.21.15.19:0/1351299395 >> v1:172.21.15.19:6789/0 conn(0x7ff390030660 legacy=0x7ff3900d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:24.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.180+0000 7ff39e2d9640 1 -- 172.21.15.19:0/1351299395 shutdown_connections 2024-03-29T11:26:24.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.180+0000 7ff39e2d9640 1 -- 172.21.15.19:0/1351299395 >> 172.21.15.19:0/1351299395 conn(0x7ff390005c00 msgr2=0x7ff390006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:24.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.184+0000 7ff39e2d9640 1 -- 172.21.15.19:0/1351299395 shutdown_connections 2024-03-29T11:26:24.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.184+0000 7ff39e2d9640 1 -- 172.21.15.19:0/1351299395 wait complete. 2024-03-29T11:26:24.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.185+0000 7ff39e2d9640 10 monclient: build_initial_monmap 2024-03-29T11:26:24.175 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.185+0000 7ff39e2d9640 10 monclient: monmap: 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.185+0000 7ff39e2d9640 1 librados: starting msgr at 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.185+0000 7ff39e2d9640 1 librados: starting objecter 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.186+0000 7ff39e2d9640 1 Processor -- start 2024-03-29T11:26:24.176 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.186+0000 7ff39e2d9640 1 -- start start 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.186+0000 7ff39e2d9640 1 librados: setting wanted keys 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.186+0000 7ff39e2d9640 1 librados: calling monclient init 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.186+0000 7ff39e2d9640 10 monclient: init 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.187+0000 7ff39e2d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.187+0000 7ff39e2d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.187+0000 7ff39e2d9640 10 monclient(hunting): picked mon.noname-c con 0x7ff390030660 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:24.177 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.187+0000 7ff39e2d9640 10 monclient(hunting): picked mon.noname-b con 0x7ff3900389e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.187+0000 7ff39e2d9640 10 monclient(hunting): picked mon.noname-a con 0x7ff3900c73d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.190+0000 7ff39e2d9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff390169a90 con 0x7ff3900c73d0 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.190+0000 7ff39e2d9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3900c6fb0 con 0x7ff3900389e0 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.190+0000 7ff39e2d9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff39016a680 con 0x7ff390030660 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.190+0000 7ff39e2d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.190+0000 7ff39e2d9640 10 monclient(hunting): authenticate will time out at 1304.454888s 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.190+0000 7ff39dad8640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff3900c73d0 0x7ff390169f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:45713/0 (socket says 172.21.15.19:45713) 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.191+0000 7ff39dad8640 1 -- 172.21.15.19:0/2544500233 learned_addr learned my addr 172.21.15.19:0/2544500233 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.192+0000 7ff38dffb640 10 client.?.objecter ms_handle_connect 0x7ff3900c73d0 2024-03-29T11:26:24.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.192+0000 7ff38dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:24.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 client.?.objecter ms_handle_connect 0x7ff3900389e0 2024-03-29T11:26:24.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 client.?.objecter ms_handle_connect 0x7ff390030660 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216763259 0 0) 0x7ff390169a90 con 0x7ff3900c73d0 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.195+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c004eb0 con 0x7ff3900c73d0 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314119378 0 0) 0x7ff39016a680 con 0x7ff390030660 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:24.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:26:24.184 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff390169a90 con 0x7ff390030660 2024-03-29T11:26:24.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695432336 0 0) 0x7ff37c004eb0 con 0x7ff3900c73d0 2024-03-29T11:26:24.185 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff37c005490 con 0x7ff3900c73d0 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff39804f400 con 0x7ff3900c73d0 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:24.186 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418442979 0 0) 0x7ff390169a90 con 0x7ff390030660 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff37c005a70 con 0x7ff390030660 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff388001830 con 0x7ff390030660 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.196+0000 7ff38dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.187 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238737685 0 0) 0x7ff3900c6fb0 con 0x7ff3900389e0 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff37c005ea0 con 0x7ff3900389e0 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3999688269 0 0) 0x7ff37c005a70 con 0x7ff390030660 2024-03-29T11:26:24.188 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 >> v1:172.21.15.62:6789/0 conn(0x7ff3900389e0 legacy=0x7ff3900c58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 >> v1:172.21.15.19:6789/0 conn(0x7ff3900c73d0 legacy=0x7ff390169f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 monclient: found mon.c 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39016a810 con 0x7ff390030660 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff388001d40 con 0x7ff390030660 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.197+0000 7ff38f7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:24.189 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.199+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff388001830 con 0x7ff390030660 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.199+0000 7ff38dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.199+0000 7ff38dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.199+0000 7ff38dffb640 10 monclient: dump: 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:24.190 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:24.191 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:24.191 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:24.191 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:24.191 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 10 monclient: _renew_subs 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3900c3130 con 0x7ff390030660 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 10 monclient: _renew_subs 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3900c35c0 con 0x7ff390030660 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.200+0000 7ff39e2d9640 1 librados: init done 2024-03-29T11:26:24.193 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.203+0000 7ff39e2d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.203+0000 7ff39e2d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.203+0000 7ff39e2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.203+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff390007b70 con 0x7ff390030660 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.205+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff388019500 con 0x7ff390030660 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff388083410 con 0x7ff390030660 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff38dffb640 10 client.4502.objecter ms_dispatch 0x7ff398000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff38dffb640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff38dffb640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff38dffb640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:26:24.194 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff39cad6640 10 monclient: get_auth_request con 0x7ff37c06e740 auth_method 0 2024-03-29T11:26:24.195 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.206+0000 7ff38dffb640 10 client.4502.objecter ms_handle_connect 0x7ff37c06e740 2024-03-29T11:26:24.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.213+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff38804ecf0 con 0x7ff390030660 2024-03-29T11:26:24.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.213+0000 7ff38dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:24.200 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.213+0000 7ff38dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:24.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.564+0000 7ff39e2d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:24.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.564+0000 7ff39e2d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:24.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.564+0000 7ff39e2d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:24.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.564+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff390005120 con 0x7ff390030660 2024-03-29T11:26:24.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.565+0000 7ff38dffb640 1 -- 172.21.15.19:0/2544500233 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff38804ecf0 con 0x7ff390030660 2024-03-29T11:26:24.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.565+0000 7ff38dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:24.552 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.565+0000 7ff38dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:24.555 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:24.555 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 10 librados: watch_flush enter 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 10 librados: watch_flush exit 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff37c06e740 legacy=0x7ff37c070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 10 monclient: shutdown 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 >> v1:172.21.15.19:6790/0 conn(0x7ff390030660 legacy=0x7ff39002fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:24.559 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 shutdown_connections 2024-03-29T11:26:24.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.572+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 >> 172.21.15.19:0/2544500233 conn(0x7ff390005c00 msgr2=0x7ff390034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:24.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.573+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 shutdown_connections 2024-03-29T11:26:24.560 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.574+0000 7ff39e2d9640 1 -- 172.21.15.19:0/2544500233 wait complete. 2024-03-29T11:26:24.561 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:24.574+0000 7ff39e2d9640 1 librados: shutdown 2024-03-29T11:26:24.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:31.386 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:31.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.552+0000 7ffbadec9640 10 monclient: get_monmap_and_config 2024-03-29T11:26:31.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.552+0000 7ffbadec9640 10 monclient: build_initial_monmap 2024-03-29T11:26:31.539 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.552+0000 7ffbadec9640 10 monclient: monmap: 2024-03-29T11:26:31.539 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:31.539 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:31.553983+0000 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:31.553983+0000 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:31.540 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.553+0000 7ffbadec9640 1 Processor -- start 2024-03-29T11:26:31.541 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.553+0000 7ffbadec9640 1 -- start start 2024-03-29T11:26:31.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.555+0000 7ffbadec9640 10 monclient: init 2024-03-29T11:26:31.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.556+0000 7ffbadec9640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:31.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.556+0000 7ffbadec9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:31.546 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.556+0000 7ffbadec9640 10 monclient(hunting): picked mon.noname-c con 0x7ffba81457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.559+0000 7ffbadec9640 10 monclient(hunting): picked mon.noname-a con 0x7ffba813f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.562+0000 7ffbadec9640 10 monclient(hunting): picked mon.noname-b con 0x7ffba80a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.562+0000 7ffbadec9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba80aaef0 con 0x7ffba813f650 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.562+0000 7ffbadec9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba8145bd0 con 0x7ffba81457f0 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.562+0000 7ffbadec9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba8145eb0 con 0x7ffba80a9d70 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.562+0000 7ffbadec9640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:31.550 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.562+0000 7ffbadec9640 10 monclient(hunting): authenticate will time out at 1311.826864s 2024-03-29T11:26:31.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.564+0000 7ffba6ffd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ffba813f650 0x7ffba80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:49667/0 (socket says 172.21.15.19:49667) 2024-03-29T11:26:31.551 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.564+0000 7ffba6ffd640 1 -- 172.21.15.19:0/1222407315 learned_addr learned my addr 172.21.15.19:0/1222407315 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.566+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793163994 0 0) 0x7ffba8145bd0 con 0x7ffba81457f0 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.566+0000 7ffba67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.566+0000 7ffba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.566+0000 7ffba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.566+0000 7ffba67fc640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.566+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb90003520 con 0x7ffba81457f0 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4162433889 0 0) 0x7ffb90003520 con 0x7ffba81457f0 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 >> v1:172.21.15.19:6790/0 conn(0x7ffba80a9d70 legacy=0x7ffba814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 >> v1:172.21.15.19:6789/0 conn(0x7ffba813f650 legacy=0x7ffba80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 monclient: found mon.noname-c 2024-03-29T11:26:31.555 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8146340 con 0x7ffba81457f0 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffba0001940 con 0x7ffba81457f0 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: dump: 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.556 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffbadec9640 5 monclient: authenticate success, global_id 4471 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffbadec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba0001b10 con 0x7ffba81457f0 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffba67fc640 1 -- 172.21.15.19:0/1222407315 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffba0001f20 con 0x7ffba81457f0 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffbadec9640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.567+0000 7ffbadec9640 4 set_mon_vals no callback set 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:31.557 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffba67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffba67fc640 10 monclient: dump: 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:31.558 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffbadec9640 10 monclient: shutdown 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffbadec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffbadec9640 1 -- 172.21.15.19:0/1222407315 >> v1:172.21.15.62:6789/0 conn(0x7ffba81457f0 legacy=0x7ffba813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffbadec9640 1 -- 172.21.15.19:0/1222407315 shutdown_connections 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.568+0000 7ffbadec9640 1 -- 172.21.15.19:0/1222407315 >> 172.21.15.19:0/1222407315 conn(0x7ffba8066a00 msgr2=0x7ffba8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.569+0000 7ffbadec9640 1 -- 172.21.15.19:0/1222407315 shutdown_connections 2024-03-29T11:26:31.579 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.569+0000 7ffbadec9640 1 -- 172.21.15.19:0/1222407315 wait complete. 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.571+0000 7ffbadec9640 10 monclient: build_initial_monmap 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.571+0000 7ffbadec9640 10 monclient: monmap: 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:31.580 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.571+0000 7ffbadec9640 1 librados: starting msgr at 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.571+0000 7ffbadec9640 1 librados: starting objecter 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.571+0000 7ffbadec9640 1 Processor -- start 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.572+0000 7ffbadec9640 1 -- start start 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.572+0000 7ffbadec9640 1 librados: setting wanted keys 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.572+0000 7ffbadec9640 1 librados: calling monclient init 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.572+0000 7ffbadec9640 10 monclient: init 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.573+0000 7ffbadec9640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.573+0000 7ffbadec9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.573+0000 7ffbadec9640 10 monclient(hunting): picked mon.noname-c con 0x7ffba80a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 10 monclient(hunting): picked mon.noname-a con 0x7ffba813f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:31.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 10 monclient(hunting): picked mon.noname-b con 0x7ffba81457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba80ae050 con 0x7ffba813f650 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba80ad9c0 con 0x7ffba81457f0 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba80adb50 con 0x7ffba80a9d70 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffbadec9640 10 monclient(hunting): authenticate will time out at 1311.840823s 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.576+0000 7ffba77fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ffba80a9d70 0x7ffba81d73d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46271/0 (socket says 172.21.15.19:46271) 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.580+0000 7ffba77fe640 1 -- 172.21.15.19:0/3420790762 learned_addr learned my addr 172.21.15.19:0/3420790762 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.580+0000 7ffb8ffff640 10 client.?.objecter ms_handle_connect 0x7ffba81457f0 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.580+0000 7ffb8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.580+0000 7ffb8ffff640 10 client.?.objecter ms_handle_connect 0x7ffba813f650 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.580+0000 7ffb8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:31.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 client.?.objecter ms_handle_connect 0x7ffba80a9d70 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739906804 0 0) 0x7ffba80adb50 con 0x7ffba80a9d70 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb94004eb0 con 0x7ffba80a9d70 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166563572 0 0) 0x7ffba80ad9c0 con 0x7ffba81457f0 2024-03-29T11:26:31.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba80adb50 con 0x7ffba81457f0 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.581+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301153806 0 0) 0x7ffb94004eb0 con 0x7ffba80a9d70 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffba80ad9c0 con 0x7ffba80a9d70 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffba0001ee0 con 0x7ffba80a9d70 2024-03-29T11:26:31.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient(hunting): dump: 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:31.585 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2659151472 0 0) 0x7ffba80ad9c0 con 0x7ffba80a9d70 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 >> v1:172.21.15.62:6789/0 conn(0x7ffba81457f0 legacy=0x7ffba80b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 >> v1:172.21.15.19:6789/0 conn(0x7ffba813f650 legacy=0x7ffba80b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 monclient: found mon.c 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:31.586 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba80adce0 con 0x7ffba80a9d70 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.582+0000 7ffb8ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffb8ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffb8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 10 monclient: _renew_subs 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba81420a0 con 0x7ffba80a9d70 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 10 monclient: _renew_subs 2024-03-29T11:26:31.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba80ad370 con 0x7ffba80a9d70 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffbadec9640 1 librados: init done 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba0003c20 con 0x7ffba80a9d70 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffb8ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffba0002100 con 0x7ffba80a9d70 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.583+0000 7ffb8ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:31.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.584+0000 7ffb8ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.584+0000 7ffb8ffff640 10 monclient: dump: 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:31.589 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.585+0000 7ffba57fa640 4 set_mon_vals no callback set 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.586+0000 7ffbadec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:31.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.586+0000 7ffbadec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.586+0000 7ffbadec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.586+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb88005080 con 0x7ffba80a9d70 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.586+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ffba0004030 con 0x7ffba80a9d70 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.587+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ffba0084d80 con 0x7ffba80a9d70 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.587+0000 7ffb8ffff640 10 client.4505.objecter ms_dispatch 0x7ffba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.587+0000 7ffb8ffff640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:31.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.587+0000 7ffb8ffff640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:31.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.587+0000 7ffb8ffff640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:26:31.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.587+0000 7ffba6ffd640 10 monclient: get_auth_request con 0x7ffb9406ded0 auth_method 0 2024-03-29T11:26:31.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.588+0000 7ffb8ffff640 10 client.4505.objecter ms_handle_connect 0x7ffb9406ded0 2024-03-29T11:26:31.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.597+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffba0050660 con 0x7ffba80a9d70 2024-03-29T11:26:31.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.597+0000 7ffb8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:31.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.597+0000 7ffb8ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:31.911 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.923+0000 7ffbadec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:31.911 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.923+0000 7ffbadec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:31.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.923+0000 7ffbadec9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:31.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.923+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb88005330 con 0x7ffba80a9d70 2024-03-29T11:26:31.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.925+0000 7ffb8ffff640 1 -- 172.21.15.19:0/3420790762 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ffba0052590 con 0x7ffba80a9d70 2024-03-29T11:26:31.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.925+0000 7ffb8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:31.912 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.925+0000 7ffb8ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:31.913 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:31.913 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:31.916 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.929+0000 7ffbadec9640 10 librados: watch_flush enter 2024-03-29T11:26:31.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 10 librados: watch_flush exit 2024-03-29T11:26:31.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:26:31.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:26:31.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 >> v1:172.21.15.19:6816/3721232097 conn(0x7ffb9406ded0 legacy=0x7ffb94070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 10 monclient: shutdown 2024-03-29T11:26:31.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:31.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.930+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 >> v1:172.21.15.19:6790/0 conn(0x7ffba80a9d70 legacy=0x7ffba81d73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:31.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.932+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 shutdown_connections 2024-03-29T11:26:31.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.932+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 >> 172.21.15.19:0/3420790762 conn(0x7ffba8066a00 msgr2=0x7ffba80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:31.919 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.933+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 shutdown_connections 2024-03-29T11:26:31.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.933+0000 7ffbadec9640 1 -- 172.21.15.19:0/3420790762 wait complete. 2024-03-29T11:26:31.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:31.934+0000 7ffbadec9640 1 librados: shutdown 2024-03-29T11:26:31.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:38.746 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:38.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.917+0000 7fe431a84640 10 monclient: get_monmap_and_config 2024-03-29T11:26:38.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.917+0000 7fe431a84640 10 monclient: build_initial_monmap 2024-03-29T11:26:38.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.917+0000 7fe431a84640 10 monclient: monmap: 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:38.917938+0000 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:38.917938+0000 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:38.905 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.907 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.918+0000 7fe431a84640 1 Processor -- start 2024-03-29T11:26:38.908 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.922+0000 7fe431a84640 1 -- start start 2024-03-29T11:26:38.908 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.922+0000 7fe431a84640 10 monclient: init 2024-03-29T11:26:38.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.923+0000 7fe431a84640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:38.909 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.923+0000 7fe431a84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 10 monclient(hunting): picked mon.noname-c con 0x7fe42c0a9ac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 10 monclient(hunting): picked mon.noname-a con 0x7fe42c13ee90 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 10 monclient(hunting): picked mon.noname-b con 0x7fe42c0af900 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0afd10 con 0x7fe42c13ee90 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0af6f0 con 0x7fe42c0a9ac0 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c0aac80 con 0x7fe42c0af900 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:38.910 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.924+0000 7fe431a84640 10 monclient(hunting): authenticate will time out at 1319.187227s 2024-03-29T11:26:38.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.929+0000 7fe42b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe42c0a9ac0 0x7fe42c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:45239/0 (socket says 172.21.15.19:45239) 2024-03-29T11:26:38.915 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.929+0000 7fe42b7fe640 1 -- 172.21.15.19:0/1664644377 learned_addr learned my addr 172.21.15.19:0/1664644377 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:38.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.930+0000 7fe42bfff640 1 -- 172.21.15.19:0/1664644377 >> v1:172.21.15.19:6790/0 conn(0x7fe42c0af900 legacy=0x7fe42c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-29T11:26:38.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.930+0000 7fe42bfff640 1 -- 172.21.15.19:0/1664644377 >> v1:172.21.15.19:6790/0 conn(0x7fe42c0af900 legacy=0x7fe42c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:26:38.917 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.930+0000 7fe42bfff640 1 --1- 172.21.15.19:0/1664644377 >> v1:172.21.15.19:6790/0 conn(0x7fe42c0af900 0x7fe42c149b00 :-1 s=OPENED pgs=667689 cs=1 l=1).handle_message read tag failed 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.930+0000 7fe42bfff640 1 --1- 172.21.15.19:0/1664644377 >> v1:172.21.15.19:6790/0 conn(0x7fe42c0af900 0x7fe42c149b00 :-1 s=OPENED pgs=667689 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.19:6790/0 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690517827 0 0) 0x7fe42c0afd10 con 0x7fe42c13ee90 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:26:38.918 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe414003db0 con 0x7fe42c13ee90 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1344871013 0 0) 0x7fe414003db0 con 0x7fe42c13ee90 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 >> v1:172.21.15.19:6790/0 conn(0x7fe42c0af900 legacy=0x7fe42c149b00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 >> v1:172.21.15.62:6789/0 conn(0x7fe42c0a9ac0 legacy=0x7fe42c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:26:38.920 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c0aaf60 con 0x7fe42c13ee90 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe4200015c0 con 0x7fe42c13ee90 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe431a84640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe431a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.931+0000 7fe42a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 10 monclient: dump: 2024-03-29T11:26:38.921 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe431a84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4200017e0 con 0x7fe42c13ee90 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:38.922 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 1 -- 172.21.15.19:0/1664644377 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe420001b70 con 0x7fe42c13ee90 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe431a84640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe431a84640 4 set_mon_vals no callback set 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.932+0000 7fe42a7fc640 10 monclient: dump: 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:38.923 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.933+0000 7fe431a84640 10 monclient: shutdown 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.933+0000 7fe431a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.933+0000 7fe431a84640 1 -- 172.21.15.19:0/1664644377 >> v1:172.21.15.19:6789/0 conn(0x7fe42c13ee90 legacy=0x7fe42c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.933+0000 7fe431a84640 1 -- 172.21.15.19:0/1664644377 shutdown_connections 2024-03-29T11:26:38.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.933+0000 7fe431a84640 1 -- 172.21.15.19:0/1664644377 >> 172.21.15.19:0/1664644377 conn(0x7fe42c0b35e0 msgr2=0x7fe42c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:38.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.934+0000 7fe431a84640 1 -- 172.21.15.19:0/1664644377 shutdown_connections 2024-03-29T11:26:38.925 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.934+0000 7fe431a84640 1 -- 172.21.15.19:0/1664644377 wait complete. 2024-03-29T11:26:38.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.935+0000 7fe431a84640 10 monclient: build_initial_monmap 2024-03-29T11:26:38.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.937+0000 7fe431a84640 10 monclient: monmap: 2024-03-29T11:26:38.926 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:38.926 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.938+0000 7fe431a84640 1 librados: starting msgr at 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.938+0000 7fe431a84640 1 librados: starting objecter 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.938+0000 7fe431a84640 1 Processor -- start 2024-03-29T11:26:38.927 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.939+0000 7fe431a84640 1 -- start start 2024-03-29T11:26:38.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.939+0000 7fe431a84640 1 librados: setting wanted keys 2024-03-29T11:26:38.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.939+0000 7fe431a84640 1 librados: calling monclient init 2024-03-29T11:26:38.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.939+0000 7fe431a84640 10 monclient: init 2024-03-29T11:26:38.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.939+0000 7fe431a84640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:38.928 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.940+0000 7fe431a84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:38.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 10 monclient(hunting): picked mon.noname-b con 0x7fe42c0a9ac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:38.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 10 monclient(hunting): picked mon.noname-a con 0x7fe42c0af900 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:38.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 10 monclient(hunting): picked mon.noname-c con 0x7fe42c13ee90 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:38.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c1dee70 con 0x7fe42c0af900 2024-03-29T11:26:38.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c1df0c0 con 0x7fe42c0a9ac0 2024-03-29T11:26:38.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe42c1dfcf0 con 0x7fe42c13ee90 2024-03-29T11:26:38.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:38.932 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.944+0000 7fe431a84640 10 monclient(hunting): authenticate will time out at 1319.207516s 2024-03-29T11:26:38.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.949+0000 7fe42b7fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fe42c0a9ac0 0x7fe42c1df5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:45357/0 (socket says 172.21.15.19:45357) 2024-03-29T11:26:38.935 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.949+0000 7fe42b7fe640 1 -- 172.21.15.19:0/1050453500 learned_addr learned my addr 172.21.15.19:0/1050453500 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:38.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.952+0000 7fe413fff640 10 client.?.objecter ms_handle_connect 0x7fe42c0a9ac0 2024-03-29T11:26:38.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:38.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 client.?.objecter ms_handle_connect 0x7fe42c0af900 2024-03-29T11:26:38.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:38.944 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 client.?.objecter ms_handle_connect 0x7fe42c13ee90 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764472800 0 0) 0x7fe42c1df0c0 con 0x7fe42c0a9ac0 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe418004eb0 con 0x7fe42c0a9ac0 2024-03-29T11:26:38.945 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267309662 0 0) 0x7fe42c1dee70 con 0x7fe42c0af900 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c1df0c0 con 0x7fe42c0af900 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028947765 0 0) 0x7fe42c1dfcf0 con 0x7fe42c13ee90 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.953+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe42c1dee70 con 0x7fe42c13ee90 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3143660021 0 0) 0x7fe42c1dee70 con 0x7fe42c13ee90 2024-03-29T11:26:38.946 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe42c1dfcf0 con 0x7fe42c13ee90 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe41c001710 con 0x7fe42c13ee90 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): dump: 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:38.947 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357075579 0 0) 0x7fe42c1df0c0 con 0x7fe42c0af900 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe42c1dee70 con 0x7fe42c0af900 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe420001af0 con 0x7fe42c0af900 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient(hunting): dump: 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:38.948 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 926532996 0 0) 0x7fe42c1dfcf0 con 0x7fe42c13ee90 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 >> v1:172.21.15.62:6789/0 conn(0x7fe42c0a9ac0 legacy=0x7fe42c1df5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 >> v1:172.21.15.19:6789/0 conn(0x7fe42c0af900 legacy=0x7fe42c071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 monclient: found mon.c 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c070dc0 con 0x7fe42c13ee90 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe413fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:26:38.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 10 monclient: _renew_subs 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe42c072020 con 0x7fe42c13ee90 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 10 monclient: _renew_subs 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe42c06e7a0 con 0x7fe42c13ee90 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.954+0000 7fe431a84640 1 librados: init done 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe41c001c20 con 0x7fe42c13ee90 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe413fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe41c003790 con 0x7fe42c13ee90 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe413fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:38.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe413fff640 10 monclient: dump: 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:38.951 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:38.952 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:38.952 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:38.952 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:38.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.956+0000 7fe429ffb640 4 set_mon_vals no callback set 2024-03-29T11:26:38.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.957+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fe41c0039b0 con 0x7fe42c13ee90 2024-03-29T11:26:38.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.958+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fe41c084870 con 0x7fe42c13ee90 2024-03-29T11:26:38.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.958+0000 7fe413fff640 10 client.4508.objecter ms_dispatch 0x7fe42c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:38.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.958+0000 7fe413fff640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.958+0000 7fe413fff640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.958+0000 7fe413fff640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.958+0000 7fe42b7fe640 10 monclient: get_auth_request con 0x7fe41806e030 auth_method 0 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.959+0000 7fe413fff640 10 client.4508.objecter ms_handle_connect 0x7fe41806e030 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.960+0000 7fe431a84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.960+0000 7fe431a84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.960+0000 7fe431a84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:38.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.960+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c005080 con 0x7fe42c13ee90 2024-03-29T11:26:38.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.968+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe41c050150 con 0x7fe42c13ee90 2024-03-29T11:26:38.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.968+0000 7fe413fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:38.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:38.968+0000 7fe413fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:39.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.295+0000 7fe431a84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:39.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.295+0000 7fe431a84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:39.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.295+0000 7fe431a84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:26:39.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.295+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe40c005330 con 0x7fe42c13ee90 2024-03-29T11:26:39.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.298+0000 7fe413fff640 1 -- 172.21.15.19:0/1050453500 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fe41c052080 con 0x7fe42c13ee90 2024-03-29T11:26:39.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.298+0000 7fe413fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:39.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.298+0000 7fe413fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:39.284 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:39.284 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:39.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 10 librados: watch_flush enter 2024-03-29T11:26:39.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 10 librados: watch_flush exit 2024-03-29T11:26:39.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:26:39.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:26:39.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 >> v1:172.21.15.19:6816/3721232097 conn(0x7fe41806e030 legacy=0x7fe4180704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:39.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 10 monclient: shutdown 2024-03-29T11:26:39.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:39.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.301+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 >> v1:172.21.15.19:6790/0 conn(0x7fe42c13ee90 legacy=0x7fe42c133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:39.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.302+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 shutdown_connections 2024-03-29T11:26:39.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.302+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 >> 172.21.15.19:0/1050453500 conn(0x7fe42c0b35e0 msgr2=0x7fe42c0ab620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:39.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.302+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 shutdown_connections 2024-03-29T11:26:39.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.303+0000 7fe431a84640 1 -- 172.21.15.19:0/1050453500 wait complete. 2024-03-29T11:26:39.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:39.303+0000 7fe431a84640 1 librados: shutdown 2024-03-29T11:26:39.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:46.115 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:46.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.280+0000 7f65fb96b640 10 monclient: get_monmap_and_config 2024-03-29T11:26:46.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.280+0000 7f65fb96b640 10 monclient: build_initial_monmap 2024-03-29T11:26:46.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.280+0000 7f65fb96b640 10 monclient: monmap: 2024-03-29T11:26:46.267 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:46.267 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:46.281700+0000 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:46.281700+0000 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:46.268 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.281+0000 7f65fb96b640 1 Processor -- start 2024-03-29T11:26:46.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.282+0000 7f65fb96b640 1 -- start start 2024-03-29T11:26:46.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.282+0000 7f65fb96b640 10 monclient: init 2024-03-29T11:26:46.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:46.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient(hunting): picked mon.noname-c con 0x7f65f406c250 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient(hunting): picked mon.noname-b con 0x7f65f4072400 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient(hunting): picked mon.noname-a con 0x7f65f406c630 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f406d760 con 0x7f65f406c630 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f4070500 con 0x7f65f406c250 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40aff50 con 0x7f65f4072400 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65fb96b640 10 monclient(hunting): authenticate will time out at 1326.546202s 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65f8f06640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f65f4072400 0x7f65f41465b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:41740/0 (socket says 172.21.15.19:41740) 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65f8f06640 1 -- 172.21.15.19:0/491015140 learned_addr learned my addr 172.21.15.19:0/491015140 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:46.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65f9707640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f65f406c250 0x7f65f4071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:42260/0 (socket says 172.21.15.19:42260) 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788026271 0 0) 0x7f65f40aff50 con 0x7f65f4072400 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e0003520 con 0x7f65f4072400 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011905785 0 0) 0x7f65f4070500 con 0x7f65f406c250 2024-03-29T11:26:46.271 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.283+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e00048a0 con 0x7f65f406c250 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.284+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826732916 0 0) 0x7f65e0003520 con 0x7f65f4072400 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.284+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 >> v1:172.21.15.62:6789/0 conn(0x7f65f406c250 legacy=0x7f65f4071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:46.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.284+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 >> v1:172.21.15.19:6789/0 conn(0x7f65f406c630 legacy=0x7f65f4151ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:46.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:46.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 1 monclient: found mon.noname-b 2024-03-29T11:26:46.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:46.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:26:46.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f40b00e0 con 0x7f65f4072400 2024-03-29T11:26:46.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:46.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:46.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f65e4001670 con 0x7f65f4072400 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: dump: 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:46.279 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65fb96b640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65fb96b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65e4001850 con 0x7f65f4072400 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 1 -- 172.21.15.19:0/491015140 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f65e4001c60 con 0x7f65f4072400 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65ebfff640 10 monclient: dump: 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:46.280 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65fb96b640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.286+0000 7f65fb96b640 4 set_mon_vals no callback set 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.287+0000 7f65fb96b640 10 monclient: shutdown 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.287+0000 7f65fb96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.287+0000 7f65fb96b640 1 -- 172.21.15.19:0/491015140 >> v1:172.21.15.19:6790/0 conn(0x7f65f4072400 legacy=0x7f65f41465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.287+0000 7f65fb96b640 1 -- 172.21.15.19:0/491015140 shutdown_connections 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.287+0000 7f65fb96b640 1 -- 172.21.15.19:0/491015140 >> 172.21.15.19:0/491015140 conn(0x7f65f4067610 msgr2=0x7f65f40b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.288+0000 7f65fb96b640 1 -- 172.21.15.19:0/491015140 shutdown_connections 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.288+0000 7f65fb96b640 1 -- 172.21.15.19:0/491015140 wait complete. 2024-03-29T11:26:46.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.288+0000 7f65fb96b640 10 monclient: build_initial_monmap 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.288+0000 7f65fb96b640 10 monclient: monmap: 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 1 librados: starting msgr at 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 1 librados: starting objecter 2024-03-29T11:26:46.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 1 Processor -- start 2024-03-29T11:26:46.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 1 -- start start 2024-03-29T11:26:46.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 1 librados: setting wanted keys 2024-03-29T11:26:46.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 1 librados: calling monclient init 2024-03-29T11:26:46.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.289+0000 7f65fb96b640 10 monclient: init 2024-03-29T11:26:46.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:46.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient(hunting): picked mon.noname-a con 0x7f65f406c250 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient(hunting): picked mon.noname-b con 0x7f65f406c630 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient(hunting): picked mon.noname-c con 0x7f65f40a94e0 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40b5400 con 0x7f65f406c250 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40b5590 con 0x7f65f406c630 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f40b5720 con 0x7f65f40a94e0 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65fb96b640 10 monclient(hunting): authenticate will time out at 1326.553355s 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65f8f06640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f65f406c630 0x7f65f413e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:44550/0 (socket says 172.21.15.19:44550) 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.290+0000 7f65f8f06640 1 -- 172.21.15.19:0/1563925403 learned_addr learned my addr 172.21.15.19:0/1563925403 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:46.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 client.?.objecter ms_handle_connect 0x7f65f406c630 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 client.?.objecter ms_handle_connect 0x7f65f40a94e0 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371967694 0 0) 0x7f65f40b5590 con 0x7f65f406c630 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.291+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65d8004eb0 con 0x7f65f406c630 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98267363 0 0) 0x7f65f40b5720 con 0x7f65f40a94e0 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:46.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f40b5590 con 0x7f65f40a94e0 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4132862518 0 0) 0x7f65d8004eb0 con 0x7f65f406c630 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f40b5720 con 0x7f65f406c630 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f65e4001b60 con 0x7f65f406c630 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.286 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:46.287 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:46.287 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:46.287 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 client.?.objecter ms_handle_connect 0x7f65f406c250 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639638408 0 0) 0x7f65f40b5590 con 0x7f65f40a94e0 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65d8004eb0 con 0x7f65f40a94e0 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f65ec001ad0 con 0x7f65f40a94e0 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.292+0000 7f65e9ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.291 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3591550016 0 0) 0x7f65f40b5720 con 0x7f65f406c630 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 >> v1:172.21.15.19:6790/0 conn(0x7f65f40a94e0 legacy=0x7f65f41e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 >> v1:172.21.15.19:6789/0 conn(0x7f65f406c250 legacy=0x7f65f41df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 1 monclient: found mon.b 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:46.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f4145da0 con 0x7f65f406c630 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 5 monclient: authenticate success, global_id 4486 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 10 monclient: _renew_subs 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f4146050 con 0x7f65f406c630 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 10 monclient: _renew_subs 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f413edd0 con 0x7f65f406c630 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.293+0000 7f65fb96b640 1 librados: init done 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65fb96b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:46.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65fb96b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65fb96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f40a98f0 con 0x7f65f406c630 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65e4001d60 con 0x7f65f406c630 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.295+0000 7f65eb7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.296+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f65e4004d90 con 0x7f65f406c630 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.296+0000 7f65e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.296+0000 7f65e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.296+0000 7f65e9ffb640 10 monclient: dump: 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:46.294 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.297+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f65e401ad60 con 0x7f65f406c630 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.298+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f65e4084a00 con 0x7f65f406c630 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.298+0000 7f65e9ffb640 10 client.4486.objecter ms_dispatch 0x7f65f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.298+0000 7f65e9ffb640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.298+0000 7f65e9ffb640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.298+0000 7f65e9ffb640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-29T11:26:46.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.298+0000 7f65f9707640 10 monclient: get_auth_request con 0x7f65d806dde0 auth_method 0 2024-03-29T11:26:46.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.299+0000 7f65e9ffb640 10 client.4486.objecter ms_handle_connect 0x7f65d806dde0 2024-03-29T11:26:46.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.303+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65e40502e0 con 0x7f65f406c630 2024-03-29T11:26:46.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.303+0000 7f65e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:46.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.303+0000 7f65e9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:46.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.616+0000 7f65fb96b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:46.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.616+0000 7f65fb96b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:46.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.616+0000 7f65fb96b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:26:46.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.616+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65f40a98f0 con 0x7f65f406c630 2024-03-29T11:26:46.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.617+0000 7f65e9ffb640 1 -- 172.21.15.19:0/1563925403 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f65f40a98f0 con 0x7f65f406c630 2024-03-29T11:26:46.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.617+0000 7f65e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:46.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.617+0000 7f65e9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:46.604 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:46.604 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:46.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.620+0000 7f65fb96b640 10 librados: watch_flush enter 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.620+0000 7f65fb96b640 10 librados: watch_flush exit 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.620+0000 7f65fb96b640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.620+0000 7f65fb96b640 10 client.4486.objecter successfully canceled tick 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 >> v1:172.21.15.19:6816/3721232097 conn(0x7f65d806dde0 legacy=0x7f65d80702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 10 monclient: shutdown 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:46.608 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 >> v1:172.21.15.62:6789/0 conn(0x7f65f406c630 legacy=0x7f65f413e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:46.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 shutdown_connections 2024-03-29T11:26:46.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 >> 172.21.15.19:0/1563925403 conn(0x7f65f4067610 msgr2=0x7f65f406cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:46.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 shutdown_connections 2024-03-29T11:26:46.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 -- 172.21.15.19:0/1563925403 wait complete. 2024-03-29T11:26:46.609 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:46.621+0000 7f65fb96b640 1 librados: shutdown 2024-03-29T11:26:46.622 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:53.429 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.594+0000 7facff5c8640 10 monclient: get_monmap_and_config 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.594+0000 7facff5c8640 10 monclient: build_initial_monmap 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.594+0000 7facff5c8640 10 monclient: monmap: 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:26:53.595041+0000 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:26:53.595041+0000 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.581 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:53.582 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:26:53.582 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:26:53.582 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.582 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.595+0000 7facff5c8640 1 Processor -- start 2024-03-29T11:26:53.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.595+0000 7facff5c8640 1 -- start start 2024-03-29T11:26:53.583 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.595+0000 7facff5c8640 10 monclient: init 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient(hunting): picked mon.noname-b con 0x7facf80b0770 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient(hunting): picked mon.noname-a con 0x7facf814a700 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient(hunting): picked mon.noname-c con 0x7facf80ac5a0 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80aba70 con 0x7facf814a700 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80aecb0 con 0x7facf80ac5a0 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf80af150 con 0x7facf80b0770 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:53.584 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facff5c8640 10 monclient(hunting): authenticate will time out at 1333.860227s 2024-03-29T11:26:53.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facfd364640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7facf80b0770 0x7facf8149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:39854/0 (socket says 172.21.15.19:39854) 2024-03-29T11:26:53.585 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.597+0000 7facfd364640 1 -- 172.21.15.19:0/2795739864 learned_addr learned my addr 172.21.15.19:0/2795739864 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:53.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958616190 0 0) 0x7facf80af150 con 0x7facf80b0770 2024-03-29T11:26:53.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:53.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:53.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:53.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:26:53.587 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7face4003520 con 0x7facf80b0770 2024-03-29T11:26:53.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181890781 0 0) 0x7facf80aecb0 con 0x7facf80ac5a0 2024-03-29T11:26:53.588 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf80af150 con 0x7facf80ac5a0 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 719245155 0 0) 0x7facf80af150 con 0x7facf80ac5a0 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 >> v1:172.21.15.19:6790/0 conn(0x7facf80b0770 legacy=0x7facf8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 >> v1:172.21.15.19:6789/0 conn(0x7facf814a700 legacy=0x7facf80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 monclient: found mon.noname-c 2024-03-29T11:26:53.589 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf80ac9d0 con 0x7facf80ac5a0 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7facf4001670 con 0x7facf80ac5a0 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: dump: 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.590 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7facff5c8640 5 monclient: authenticate success, global_id 4489 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7facff5c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facf4001850 con 0x7facf80ac5a0 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7facff5c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7facff5c8640 4 set_mon_vals no callback set 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 1 -- 172.21.15.19:0/2795739864 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7facf4001ca0 con 0x7facf80ac5a0 2024-03-29T11:26:53.591 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.598+0000 7faceffff640 10 monclient: dump: 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.592 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 10 monclient: shutdown 2024-03-29T11:26:53.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:53.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 1 -- 172.21.15.19:0/2795739864 >> v1:172.21.15.62:6789/0 conn(0x7facf80ac5a0 legacy=0x7facf8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 1 -- 172.21.15.19:0/2795739864 shutdown_connections 2024-03-29T11:26:53.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 1 -- 172.21.15.19:0/2795739864 >> 172.21.15.19:0/2795739864 conn(0x7facf806b660 msgr2=0x7facf806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:53.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 1 -- 172.21.15.19:0/2795739864 shutdown_connections 2024-03-29T11:26:53.593 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.600+0000 7facff5c8640 1 -- 172.21.15.19:0/2795739864 wait complete. 2024-03-29T11:26:53.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.602+0000 7facff5c8640 10 monclient: build_initial_monmap 2024-03-29T11:26:53.595 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.602+0000 7facff5c8640 10 monclient: monmap: 2024-03-29T11:26:53.595 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:26:53.595 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.602+0000 7facff5c8640 1 librados: starting msgr at 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.602+0000 7facff5c8640 1 librados: starting objecter 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.602+0000 7facff5c8640 1 Processor -- start 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.603+0000 7facff5c8640 1 -- start start 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.603+0000 7facff5c8640 1 librados: setting wanted keys 2024-03-29T11:26:53.596 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.603+0000 7facff5c8640 1 librados: calling monclient init 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.603+0000 7facff5c8640 10 monclient: init 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient(hunting): picked mon.noname-c con 0x7facf80ac5a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient(hunting): picked mon.noname-a con 0x7facf80b0770 addr v1:172.21.15.19:6789/0 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient(hunting): picked mon.noname-b con 0x7facf814a700 addr v1:172.21.15.62:6789/0 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf81db360 con 0x7facf80b0770 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf81239d0 con 0x7facf814a700 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf8145a80 con 0x7facf80ac5a0 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facff5c8640 10 monclient(hunting): authenticate will time out at 1333.867400s 2024-03-29T11:26:53.597 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facfcb63640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7facf80b0770 0x7facf81266e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:32778/0 (socket says 172.21.15.19:32778) 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facfcb63640 1 -- 172.21.15.19:0/3005531858 learned_addr learned my addr 172.21.15.19:0/3005531858 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facedffb640 10 client.?.objecter ms_handle_connect 0x7facf80b0770 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facedffb640 10 client.?.objecter ms_handle_connect 0x7facf814a700 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facedffb640 10 client.?.objecter ms_handle_connect 0x7facf80ac5a0 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.604+0000 7facedffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695574430 0 0) 0x7facf81db360 con 0x7facf80b0770 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:53.598 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facdc004eb0 con 0x7facf80b0770 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148033158 0 0) 0x7facf81239d0 con 0x7facf814a700 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf81db360 con 0x7facf814a700 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438504750 0 0) 0x7facf8145a80 con 0x7facf80ac5a0 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:26:53.599 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.605+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facdc005360 con 0x7facf80ac5a0 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2096658578 0 0) 0x7facdc004eb0 con 0x7facf80b0770 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facf8145a80 con 0x7facf80b0770 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7facf0001d10 con 0x7facf80b0770 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): dump: 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.600 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 223661329 0 0) 0x7facf81db360 con 0x7facf814a700 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facdc004eb0 con 0x7facf814a700 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7facf4001ca0 con 0x7facf814a700 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.606+0000 7facedffb640 10 monclient(hunting): dump: 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.601 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2546274413 0 0) 0x7facf8145a80 con 0x7facf80b0770 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 >> v1:172.21.15.19:6790/0 conn(0x7facf80ac5a0 legacy=0x7facf81db880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 >> v1:172.21.15.62:6789/0 conn(0x7facf814a700 legacy=0x7facf8144d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 1 monclient: found mon.a 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:53.602 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8145c10 con 0x7facf80b0770 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 10 monclient: _renew_subs 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf81dbf90 con 0x7facf80b0770 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 10 monclient: _renew_subs 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facf8126fd0 con 0x7facf80b0770 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.607+0000 7facff5c8640 1 librados: init done 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facf0003990 con 0x7facf80b0770 2024-03-29T11:26:53.603 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7facf0003ce0 con 0x7facf80b0770 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facedffb640 10 monclient: dump: 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:26:53.604 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:26:53.605 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.608+0000 7facef7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.609+0000 7facff5c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.610+0000 7facff5c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.610+0000 7facff5c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.610+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd8005080 con 0x7facf80b0770 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.610+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7facf0003e40 con 0x7facf80b0770 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.611+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7facf0084d50 con 0x7facf80b0770 2024-03-29T11:26:53.606 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.611+0000 7facedffb640 10 client.4494.objecter ms_dispatch 0x7facf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:53.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.611+0000 7facedffb640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:53.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.611+0000 7facedffb640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:53.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.611+0000 7facedffb640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-29T11:26:53.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.611+0000 7facfd364640 10 monclient: get_auth_request con 0x7facdc06e2b0 auth_method 0 2024-03-29T11:26:53.607 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.612+0000 7facedffb640 10 client.4494.objecter ms_handle_connect 0x7facdc06e2b0 2024-03-29T11:26:53.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.625+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facf0050690 con 0x7facf80b0770 2024-03-29T11:26:53.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.625+0000 7facedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:53.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:53.625+0000 7facedffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:53.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.003+0000 7facff5c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:53.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.003+0000 7facff5c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:53.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.003+0000 7facff5c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:26:53.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.003+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facd8005330 con 0x7facf80b0770 2024-03-29T11:26:53.993 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:26:53.994 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:53.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.006+0000 7facedffb640 1 -- 172.21.15.19:0/3005531858 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7facf0084400 con 0x7facf80b0770 2024-03-29T11:26:53.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.006+0000 7facedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:53.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.006+0000 7facedffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 10 librados: watch_flush enter 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 10 librados: watch_flush exit 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 10 client.4494.objecter successfully canceled tick 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 >> v1:172.21.15.19:6816/3721232097 conn(0x7facdc06e2b0 legacy=0x7facdc070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 10 monclient: shutdown 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 >> v1:172.21.15.19:6789/0 conn(0x7facf80b0770 legacy=0x7facf81266e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:53.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.010+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 shutdown_connections 2024-03-29T11:26:53.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.011+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 >> 172.21.15.19:0/3005531858 conn(0x7facf806b660 msgr2=0x7facf80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:53.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.011+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 shutdown_connections 2024-03-29T11:26:53.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.011+0000 7facff5c8640 1 -- 172.21.15.19:0/3005531858 wait complete. 2024-03-29T11:26:53.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:26:54.011+0000 7facff5c8640 1 librados: shutdown 2024-03-29T11:26:54.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:00.820 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:00.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.987+0000 7f5e1b001640 10 monclient: get_monmap_and_config 2024-03-29T11:27:00.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.987+0000 7f5e1b001640 10 monclient: build_initial_monmap 2024-03-29T11:27:00.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.987+0000 7f5e1b001640 10 monclient: monmap: 2024-03-29T11:27:00.974 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:00.974 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:00.988031+0000 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:00.988031+0000 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:00.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.988+0000 7f5e1b001640 1 Processor -- start 2024-03-29T11:27:00.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.989+0000 7f5e1b001640 1 -- start start 2024-03-29T11:27:00.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.989+0000 7f5e1b001640 10 monclient: init 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0c006d80 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0c0cc310 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0c0c9430 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c008470 con 0x7f5e0c0cc310 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c02be30 con 0x7f5e0c006d80 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c02c110 con 0x7f5e0c0c9430 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:00.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.990+0000 7f5e1b001640 10 monclient(hunting): authenticate will time out at 1341.253297s 2024-03-29T11:27:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.993+0000 7f5e19fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f5e0c006d80 0x7f5e0c0cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:45885/0 (socket says 172.21.15.19:45885) 2024-03-29T11:27:00.980 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.993+0000 7f5e19fff640 1 -- 172.21.15.19:0/3924149155 learned_addr learned my addr 172.21.15.19:0/3924149155 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:00.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.995+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728410239 0 0) 0x7f5e0c02be30 con 0x7f5e0c006d80 2024-03-29T11:27:00.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.996+0000 7f5e18ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.996+0000 7f5e18ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.996+0000 7f5e18ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.996+0000 7f5e18ffd640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:27:00.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.996+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e00003520 con 0x7f5e0c006d80 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619580067 0 0) 0x7f5e0c008470 con 0x7f5e0c0cc310 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e0c02be30 con 0x7f5e0c0cc310 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445855125 0 0) 0x7f5e0c02be30 con 0x7f5e0c0cc310 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 >> v1:172.21.15.19:6790/0 conn(0x7f5e0c0c9430 legacy=0x7f5e0c0d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 >> v1:172.21.15.62:6789/0 conn(0x7f5e0c006d80 legacy=0x7f5e0c0cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.987 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 monclient: found mon.noname-a 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c02c3f0 con 0x7f5e0c0cc310 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e1b001640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e1b001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5e100015a0 con 0x7f5e0c0cc310 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.997+0000 7f5e18ffd640 10 monclient: dump: 2024-03-29T11:27:00.988 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e1b001640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e100017c0 con 0x7f5e0c0cc310 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e18ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e1b001640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e1b001640 4 set_mon_vals no callback set 2024-03-29T11:27:00.989 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e18ffd640 1 -- 172.21.15.19:0/3924149155 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5e10001ad0 con 0x7f5e0c0cc310 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e18ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e18ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e18ffd640 10 monclient: dump: 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:00.990 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e1b001640 10 monclient: shutdown 2024-03-29T11:27:00.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e1b001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:00.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.998+0000 7f5e1b001640 1 -- 172.21.15.19:0/3924149155 >> v1:172.21.15.19:6789/0 conn(0x7f5e0c0cc310 legacy=0x7f5e0c0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.999+0000 7f5e1b001640 1 -- 172.21.15.19:0/3924149155 shutdown_connections 2024-03-29T11:27:00.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.999+0000 7f5e1b001640 1 -- 172.21.15.19:0/3924149155 >> 172.21.15.19:0/3924149155 conn(0x7f5e0c005c00 msgr2=0x7f5e0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:00.991 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.999+0000 7f5e1b001640 1 -- 172.21.15.19:0/3924149155 shutdown_connections 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:00.999+0000 7f5e1b001640 1 -- 172.21.15.19:0/3924149155 wait complete. 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 10 monclient: build_initial_monmap 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 10 monclient: monmap: 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:00.994 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 1 librados: starting msgr at 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 1 librados: starting objecter 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 1 Processor -- start 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 1 -- start start 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 1 librados: setting wanted keys 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 1 librados: calling monclient init 2024-03-29T11:27:00.995 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.000+0000 7f5e1b001640 10 monclient: init 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient(hunting): picked mon.noname-a con 0x7f5e0c006d80 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0c0c9430 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0c0cc310 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c156d90 con 0x7f5e0c006d80 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c02d7c0 con 0x7f5e0c0cc310 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c02d950 con 0x7f5e0c0c9430 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.001+0000 7f5e1b001640 10 monclient(hunting): authenticate will time out at 1341.264519s 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.002+0000 7f5e19fff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f5e0c006d80 0x7f5e0c02d0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:36398/0 (socket says 172.21.15.19:36398) 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.002+0000 7f5e19fff640 1 -- 172.21.15.19:0/3254419623 learned_addr learned my addr 172.21.15.19:0/3254419623 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 client.?.objecter ms_handle_connect 0x7f5e0c006d80 2024-03-29T11:27:00.996 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716735690 0 0) 0x7f5e0c156d90 con 0x7f5e0c006d80 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df8004eb0 con 0x7f5e0c006d80 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268908777 0 0) 0x7f5df8004eb0 con 0x7f5e0c006d80 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e0c156d90 con 0x7f5e0c006d80 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5e04001390 con 0x7f5e0c006d80 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.003+0000 7f5e0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.004+0000 7f5e0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.004+0000 7f5e0a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:00.997 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.004+0000 7f5e0a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.004+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1237421580 0 0) 0x7f5e0c156d90 con 0x7f5e0c006d80 2024-03-29T11:27:00.998 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.004+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 >> v1:172.21.15.19:6790/0 conn(0x7f5e0c0c9430 legacy=0x7f5e0c0c5810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 >> v1:172.21.15.62:6789/0 conn(0x7f5e0c0cc310 legacy=0x7f5e0c1641c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 1 monclient: found mon.a 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c02c8b0 con 0x7f5e0c006d80 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:00.999 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 client.?.objecter ms_handle_connect 0x7f5e0c0cc310 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e04001670 con 0x7f5e0c006d80 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f5e04004c30 con 0x7f5e0c006d80 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:01.000 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e0a7fc640 10 monclient: dump: 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:01.001 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:01.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:27:01.002 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 10 monclient: _renew_subs 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c02cb80 con 0x7f5e0c006d80 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 10 monclient: _renew_subs 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c0c6100 con 0x7f5e0c006d80 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.006+0000 7f5e1b001640 1 librados: init done 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.008+0000 7f5e0bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:01.003 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.010+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f5e0401ac00 con 0x7f5e0c006d80 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.010+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f5e04084ba0 con 0x7f5e0c006d80 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.010+0000 7f5e0a7fc640 10 client.4500.objecter ms_dispatch 0x7f5e14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e0a7fc640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e0a7fc640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e0a7fc640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e1b001640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e1b001640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e1b001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc005080 con 0x7f5e0c006d80 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e197fe640 10 monclient: get_auth_request con 0x7f5df806db20 auth_method 0 2024-03-29T11:27:01.004 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.011+0000 7f5e0a7fc640 10 client.4500.objecter ms_handle_connect 0x7f5df806db20 2024-03-29T11:27:01.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.022+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e040502b0 con 0x7f5e0c006d80 2024-03-29T11:27:01.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.022+0000 7f5e0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:01.012 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.022+0000 7f5e0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:01.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.370+0000 7f5e1b001640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:01.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.370+0000 7f5e1b001640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:01.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.370+0000 7f5e1b001640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:01.357 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.370+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dfc005330 con 0x7f5e0c006d80 2024-03-29T11:27:01.358 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.371+0000 7f5e0a7fc640 1 -- 172.21.15.19:0/3254419623 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f5e040523b0 con 0x7f5e0c006d80 2024-03-29T11:27:01.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.371+0000 7f5e0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:01.359 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.371+0000 7f5e0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:01.359 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:01.359 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:01.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 10 librados: watch_flush enter 2024-03-29T11:27:01.362 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 10 librados: watch_flush exit 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 >> v1:172.21.15.19:6816/3721232097 conn(0x7f5df806db20 legacy=0x7f5df806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 10 monclient: shutdown 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 >> v1:172.21.15.19:6789/0 conn(0x7f5e0c006d80 legacy=0x7f5e0c02d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.375+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 shutdown_connections 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.376+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 >> 172.21.15.19:0/3254419623 conn(0x7f5e0c005c00 msgr2=0x7f5e0c039c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:01.363 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.376+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 shutdown_connections 2024-03-29T11:27:01.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.376+0000 7f5e1b001640 1 -- 172.21.15.19:0/3254419623 wait complete. 2024-03-29T11:27:01.364 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:01.376+0000 7f5e1b001640 1 librados: shutdown 2024-03-29T11:27:01.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:08.186 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:08.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.334+0000 7fa3e6764640 10 monclient: get_monmap_and_config 2024-03-29T11:27:08.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.334+0000 7fa3e6764640 10 monclient: build_initial_monmap 2024-03-29T11:27:08.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.335+0000 7fa3e6764640 10 monclient: monmap: 2024-03-29T11:27:08.322 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:08.322 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:08.322 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:08.335780+0000 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:08.335780+0000 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:08.323 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:08.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.337+0000 7fa3e6764640 1 Processor -- start 2024-03-29T11:27:08.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.337+0000 7fa3e6764640 1 -- start start 2024-03-29T11:27:08.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.337+0000 7fa3e6764640 10 monclient: init 2024-03-29T11:27:08.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:08.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:08.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e01457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:08.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e013f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:08.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e00a9d70 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:08.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e00aaef0 con 0x7fa3e00a9d70 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e0145bd0 con 0x7fa3e013f650 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e0145eb0 con 0x7fa3e01457f0 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3e6764640 10 monclient(hunting): authenticate will time out at 1348.601288s 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3dffff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa3e01457f0 0x7fa3e013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:56964/0 (socket says 172.21.15.19:56964) 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3dffff640 1 -- 172.21.15.19:0/2468598777 learned_addr learned my addr 172.21.15.19:0/2468598777 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224355096 0 0) 0x7fa3e0145eb0 con 0x7fa3e01457f0 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3deffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3deffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3deffd640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.338+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3c8003520 con 0x7fa3e01457f0 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704593144 0 0) 0x7fa3c8003520 con 0x7fa3e01457f0 2024-03-29T11:27:08.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 >> v1:172.21.15.62:6789/0 conn(0x7fa3e013f650 legacy=0x7fa3e00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:08.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 >> v1:172.21.15.19:6789/0 conn(0x7fa3e00a9d70 legacy=0x7fa3e014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0146340 con 0x7fa3e01457f0 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.339+0000 7fa3deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3e6764640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3e6764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:08.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa3d4001690 con 0x7fa3e01457f0 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: dump: 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:08.330 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3e6764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d4001920 con 0x7fa3e01457f0 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3e6764640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3e6764640 4 set_mon_vals no callback set 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 1 -- 172.21.15.19:0/2468598777 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa3d4001cb0 con 0x7fa3e01457f0 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.340+0000 7fa3deffd640 10 monclient: dump: 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:08.331 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 10 monclient: shutdown 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 1 -- 172.21.15.19:0/2468598777 >> v1:172.21.15.19:6790/0 conn(0x7fa3e01457f0 legacy=0x7fa3e013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 1 -- 172.21.15.19:0/2468598777 shutdown_connections 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 1 -- 172.21.15.19:0/2468598777 >> 172.21.15.19:0/2468598777 conn(0x7fa3e0066a00 msgr2=0x7fa3e0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 1 -- 172.21.15.19:0/2468598777 shutdown_connections 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.341+0000 7fa3e6764640 1 -- 172.21.15.19:0/2468598777 wait complete. 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 10 monclient: build_initial_monmap 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 10 monclient: monmap: 2024-03-29T11:27:08.332 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:08.333 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 1 librados: starting msgr at 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 1 librados: starting objecter 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 1 Processor -- start 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 1 -- start start 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 1 librados: setting wanted keys 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 1 librados: calling monclient init 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.342+0000 7fa3e6764640 10 monclient: init 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:08.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e00a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e013f650 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e01457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e01d7780 con 0x7fa3e01457f0 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e01d7910 con 0x7fa3e013f650 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e0142c60 con 0x7fa3e00a9d70 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.343+0000 7fa3e6764640 10 monclient(hunting): authenticate will time out at 1348.606701s 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dffff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa3e00a9d70 0x7fa3e01d8080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:57484/0 (socket says 172.21.15.19:57484) 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dffff640 1 -- 172.21.15.19:0/3766985550 learned_addr learned my addr 172.21.15.19:0/3766985550 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 client.?.objecter ms_handle_connect 0x7fa3e00a9d70 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175348252 0 0) 0x7fa3e0142c60 con 0x7fa3e00a9d70 2024-03-29T11:27:08.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc004eb0 con 0x7fa3e00a9d70 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 client.?.objecter ms_handle_connect 0x7fa3e013f650 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.344+0000 7fa3dcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4288998219 0 0) 0x7fa3cc004eb0 con 0x7fa3e00a9d70 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3e0142c60 con 0x7fa3e00a9d70 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa3d4001610 con 0x7fa3e00a9d70 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 10 monclient(hunting): dump: 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:08.339 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3891198476 0 0) 0x7fa3e0142c60 con 0x7fa3e00a9d70 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 >> v1:172.21.15.62:6789/0 conn(0x7fa3e013f650 legacy=0x7fa3e0136600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.345+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 >> v1:172.21.15.19:6789/0 conn(0x7fa3e01457f0 legacy=0x7fa3e0133c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:08.340 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 1 monclient: found mon.c 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0142df0 con 0x7fa3e00a9d70 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3dcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 10 monclient: _renew_subs 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e0134c50 con 0x7fa3e00a9d70 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 10 monclient: _renew_subs 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:08.341 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e01426c0 con 0x7fa3e00a9d70 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.347+0000 7fa3e6764640 1 librados: init done 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.348+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d4004a90 con 0x7fa3e00a9d70 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.348+0000 7fa3dcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.348+0000 7fa3de7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa3d4004da0 con 0x7fa3e00a9d70 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3dcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3dcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3dcff9640 10 monclient: dump: 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:08.342 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3e6764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3e6764640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3e6764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:08.343 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.349+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c0005080 con 0x7fa3e00a9d70 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fa3d401ad70 con 0x7fa3e00a9d70 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fa3d4084c00 con 0x7fa3e00a9d70 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3dcff9640 10 client.4529.objecter ms_dispatch 0x7fa3e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3dcff9640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3dcff9640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3df7fe640 10 monclient: get_auth_request con 0x7fa3cc06dac0 auth_method 0 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.350+0000 7fa3dcff9640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T11:27:08.344 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.351+0000 7fa3dcff9640 10 client.4529.objecter ms_handle_connect 0x7fa3cc06dac0 2024-03-29T11:27:08.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.357+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa3d4050540 con 0x7fa3e00a9d70 2024-03-29T11:27:08.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.357+0000 7fa3dcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:08.345 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.357+0000 7fa3dcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:08.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.675+0000 7fa3e6764640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:08.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.675+0000 7fa3e6764640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:08.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.675+0000 7fa3e6764640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:08.663 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.675+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3c0005330 con 0x7fa3e00a9d70 2024-03-29T11:27:08.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.676+0000 7fa3dcff9640 1 -- 172.21.15.19:0/3766985550 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fa3d4050540 con 0x7fa3e00a9d70 2024-03-29T11:27:08.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.676+0000 7fa3dcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:08.664 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.676+0000 7fa3dcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:08.664 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:08.664 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 10 librados: watch_flush enter 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 10 librados: watch_flush exit 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 10 client.4529.objecter successfully canceled tick 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 >> v1:172.21.15.19:6816/3721232097 conn(0x7fa3cc06dac0 legacy=0x7fa3cc06ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 10 monclient: shutdown 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 >> v1:172.21.15.19:6790/0 conn(0x7fa3e00a9d70 legacy=0x7fa3e01d8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 shutdown_connections 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 >> 172.21.15.19:0/3766985550 conn(0x7fa3e0066a00 msgr2=0x7fa3e00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:08.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 shutdown_connections 2024-03-29T11:27:08.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.680+0000 7fa3e6764640 1 -- 172.21.15.19:0/3766985550 wait complete. 2024-03-29T11:27:08.669 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:08.681+0000 7fa3e6764640 1 librados: shutdown 2024-03-29T11:27:08.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:15.490 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:15.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.637+0000 7f0fa280f640 10 monclient: get_monmap_and_config 2024-03-29T11:27:15.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.637+0000 7f0fa280f640 10 monclient: build_initial_monmap 2024-03-29T11:27:15.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.637+0000 7f0fa280f640 10 monclient: monmap: 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:15.638725+0000 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:15.638725+0000 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:15.626 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:15.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 1 Processor -- start 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 1 -- start start 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient: init 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c0b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c0ad850 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c13c1b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0ac8e0 con 0x7f0f9c0ad850 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0ad330 con 0x7f0f9c0b0770 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c0acfe0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.639+0000 7f0fa280f640 10 monclient(hunting): authenticate will time out at 1355.902955s 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0fa0dac640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f0f9c13c1b0 0x7f0f9c14a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48230/0 (socket says 172.21.15.19:48230) 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0fa0dac640 1 -- 172.21.15.19:0/3128368914 learned_addr learned my addr 172.21.15.19:0/3128368914 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9bfff640 1 --1- 172.21.15.19:0/3128368914 >> v1:172.21.15.62:6789/0 conn(0x7f0f9c0b0770 0x7f0f9c072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:37576/0 (socket says 172.21.15.19:37576) 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076259659 0 0) 0x7f0f9c0acfe0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f84003520 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282642752 0 0) 0x7f0f9c0ad330 con 0x7f0f9c0b0770 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:15.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:15.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:15.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:27:15.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f9c0acfe0 con 0x7f0f9c0b0770 2024-03-29T11:27:15.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3471504394 0 0) 0x7f0f84003520 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 >> v1:172.21.15.62:6789/0 conn(0x7f0f9c0b0770 legacy=0x7f0f9c072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.640+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 >> v1:172.21.15.19:6789/0 conn(0x7f0f9c0ad850 legacy=0x7f0f9c0aaae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 1 monclient: found mon.noname-b 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c0ab1f0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: _finish_auth 0 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0f94001690 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:15.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0f9affd640 10 monclient: dump: 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0fa280f640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.643+0000 7f0fa280f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:15.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94001870 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0f9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0f9affd640 1 -- 172.21.15.19:0/3128368914 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0f94001c80 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0f9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0f9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0f9affd640 10 monclient: dump: 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:15.639 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0fa280f640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0fa280f640 4 set_mon_vals no callback set 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0fa280f640 10 monclient: shutdown 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0fa280f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.644+0000 7f0fa280f640 1 -- 172.21.15.19:0/3128368914 >> v1:172.21.15.19:6790/0 conn(0x7f0f9c13c1b0 legacy=0x7f0f9c14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.645+0000 7f0fa280f640 1 -- 172.21.15.19:0/3128368914 shutdown_connections 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.645+0000 7f0fa280f640 1 -- 172.21.15.19:0/3128368914 >> 172.21.15.19:0/3128368914 conn(0x7f0f9c06b660 msgr2=0x7f0f9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.645+0000 7f0fa280f640 1 -- 172.21.15.19:0/3128368914 shutdown_connections 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.645+0000 7f0fa280f640 1 -- 172.21.15.19:0/3128368914 wait complete. 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.646+0000 7f0fa280f640 10 monclient: build_initial_monmap 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.646+0000 7f0fa280f640 10 monclient: monmap: 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:15.640 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:15.641 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.646+0000 7f0fa280f640 1 librados: starting msgr at 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.646+0000 7f0fa280f640 1 librados: starting objecter 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.647+0000 7f0fa280f640 1 Processor -- start 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.647+0000 7f0fa280f640 1 -- start start 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.647+0000 7f0fa280f640 1 librados: setting wanted keys 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.647+0000 7f0fa280f640 1 librados: calling monclient init 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.647+0000 7f0fa280f640 10 monclient: init 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient(hunting): picked mon.noname-a con 0x7f0f9c0ad850 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient(hunting): picked mon.noname-b con 0x7f0f9c0b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient(hunting): picked mon.noname-c con 0x7f0f9c13c1b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c13f120 con 0x7f0f9c0ad850 2024-03-29T11:27:15.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c12d380 con 0x7f0f9c0b0770 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa0dac640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f0f9c13c1b0 0x7f0f9c12cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:50066/0 (socket says 172.21.15.19:50066) 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa0dac640 1 -- 172.21.15.19:0/1399822127 learned_addr learned my addr 172.21.15.19:0/1399822127 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f9c12e870 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0f98ff9640 10 client.?.objecter ms_handle_connect 0x7f0f9c13c1b0 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0f98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0f98ff9640 10 client.?.objecter ms_handle_connect 0x7f0f9c0b0770 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0f98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0fa280f640 10 monclient(hunting): authenticate will time out at 1355.911875s 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.648+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580512356 0 0) 0x7f0f9c12e870 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:15.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f88004eb0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2968749990 0 0) 0x7f0f88004eb0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f9c12e870 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0f94001c40 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:15.644 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1666889682 0 0) 0x7f0f9c12e870 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 >> v1:172.21.15.62:6789/0 conn(0x7f0f9c0b0770 legacy=0x7f0f9c13ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.649+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 >> v1:172.21.15.19:6789/0 conn(0x7f0f9c0ad850 legacy=0x7f0f9c12c560 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:15.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:15.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 1 monclient: found mon.c 2024-03-29T11:27:15.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:15.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:15.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c1309d0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0f98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0fa280f640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0fa280f640 10 monclient: _renew_subs 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0fa280f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.651+0000 7f0fa280f640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c130cb0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.652+0000 7f0fa280f640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.652+0000 7f0fa280f640 10 monclient: _renew_subs 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.652+0000 7f0fa280f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.652+0000 7f0fa280f640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c12e500 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.652+0000 7f0fa280f640 1 librados: init done 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f94004b00 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0f94004f50 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f98ff9640 10 monclient: dump: 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:15.650 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.653+0000 7f0f99ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.654+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f0f940050b0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.654+0000 7f0f827fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:15.651 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.654+0000 7f0f827fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.654+0000 7f0f827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.654+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c075f10 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.655+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f0f94084bf0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.655+0000 7f0f98ff9640 10 client.4535.objecter ms_dispatch 0x7f0f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.655+0000 7f0f98ff9640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.655+0000 7f0f98ff9640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.655+0000 7f0f9bfff640 10 monclient: get_auth_request con 0x7f0f8806dd50 auth_method 0 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.655+0000 7f0f98ff9640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:27:15.652 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.656+0000 7f0f98ff9640 10 client.4535.objecter ms_handle_connect 0x7f0f8806dd50 2024-03-29T11:27:15.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.663+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f94050610 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.663+0000 7f0f98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:15.653 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.663+0000 7f0f98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:15.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.979+0000 7f0f827fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:15.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.979+0000 7f0f827fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:15.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.979+0000 7f0f827fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:15.967 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.979+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f9c0aeac0 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.980+0000 7f0f98ff9640 1 -- 172.21.15.19:0/1399822127 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f0f94052540 con 0x7f0f9c13c1b0 2024-03-29T11:27:15.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.980+0000 7f0f98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:15.968 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.980+0000 7f0f98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:15.968 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:15.968 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:15.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.983+0000 7f0f827fc640 10 librados: watch_flush enter 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 10 librados: watch_flush exit 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 >> v1:172.21.15.19:6816/3721232097 conn(0x7f0f8806dd50 legacy=0x7f0f88070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 10 monclient: shutdown 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 >> v1:172.21.15.19:6790/0 conn(0x7f0f9c13c1b0 legacy=0x7f0f9c12cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 shutdown_connections 2024-03-29T11:27:15.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 >> 172.21.15.19:0/1399822127 conn(0x7f0f9c06b660 msgr2=0x7f0f9c0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:15.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.984+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 shutdown_connections 2024-03-29T11:27:15.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.985+0000 7f0f827fc640 1 -- 172.21.15.19:0/1399822127 wait complete. 2024-03-29T11:27:15.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:15.985+0000 7f0f827fc640 1 librados: shutdown 2024-03-29T11:27:15.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:22.796 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.953+0000 7f770190e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.953+0000 7f770190e640 10 monclient: build_initial_monmap 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.953+0000 7f770190e640 10 monclient: monmap: 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:22.954376+0000 2024-03-29T11:27:22.941 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:22.954376+0000 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.954+0000 7f770190e640 1 Processor -- start 2024-03-29T11:27:22.942 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.954+0000 7f770190e640 1 -- start start 2024-03-29T11:27:22.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.954+0000 7f770190e640 10 monclient: init 2024-03-29T11:27:22.943 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient(hunting): picked mon.noname-c con 0x7f76fc1457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient(hunting): picked mon.noname-b con 0x7f76fc13f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient(hunting): picked mon.noname-a con 0x7f76fc0a9d70 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc0aaef0 con 0x7f76fc0a9d70 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc145bd0 con 0x7f76fc1457f0 2024-03-29T11:27:22.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc145eb0 con 0x7f76fc13f650 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.955+0000 7f770190e640 10 monclient(hunting): authenticate will time out at 1363.218841s 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.956+0000 7f76f3fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f76fc13f650 0x7f76fc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:40732/0 (socket says 172.21.15.19:40732) 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.956+0000 7f76f3fff640 1 -- 172.21.15.19:0/1589821046 learned_addr learned my addr 172.21.15.19:0/1589821046 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604191549 0 0) 0x7f76fc145eb0 con 0x7f76fc13f650 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e8003520 con 0x7f76fc13f650 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 573277276 0 0) 0x7f76e8003520 con 0x7f76fc13f650 2024-03-29T11:27:22.950 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.959+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 >> v1:172.21.15.62:6789/0 conn(0x7f76fc1457f0 legacy=0x7f76fc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 >> v1:172.21.15.19:6789/0 conn(0x7f76fc0a9d70 legacy=0x7f76fc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc146340 con 0x7f76fc13f650 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f76e4001690 con 0x7f76fc13f650 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:22.952 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f76fa7fc640 10 monclient: dump: 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f770190e640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.962+0000 7f770190e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e4001920 con 0x7f76fc13f650 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f76fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:22.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f770190e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f770190e640 4 set_mon_vals no callback set 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f76fa7fc640 1 -- 172.21.15.19:0/1589821046 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f76e4001d30 con 0x7f76fc13f650 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f76fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f76fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f76fa7fc640 10 monclient: dump: 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:22.954 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f770190e640 10 monclient: shutdown 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f770190e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.963+0000 7f770190e640 1 -- 172.21.15.19:0/1589821046 >> v1:172.21.15.19:6790/0 conn(0x7f76fc13f650 legacy=0x7f76fc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.964+0000 7f770190e640 1 -- 172.21.15.19:0/1589821046 shutdown_connections 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.964+0000 7f770190e640 1 -- 172.21.15.19:0/1589821046 >> 172.21.15.19:0/1589821046 conn(0x7f76fc066a00 msgr2=0x7f76fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.964+0000 7f770190e640 1 -- 172.21.15.19:0/1589821046 shutdown_connections 2024-03-29T11:27:22.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.964+0000 7f770190e640 1 -- 172.21.15.19:0/1589821046 wait complete. 2024-03-29T11:27:22.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.965+0000 7f770190e640 10 monclient: build_initial_monmap 2024-03-29T11:27:22.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.965+0000 7f770190e640 10 monclient: monmap: 2024-03-29T11:27:22.956 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:22.956 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:22.956 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.965+0000 7f770190e640 1 librados: starting msgr at 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.965+0000 7f770190e640 1 librados: starting objecter 2024-03-29T11:27:22.957 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.965+0000 7f770190e640 1 Processor -- start 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 1 -- start start 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 1 librados: setting wanted keys 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 1 librados: calling monclient init 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient: init 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient(hunting): picked mon.noname-c con 0x7f76fc0a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient(hunting): picked mon.noname-a con 0x7f76fc13f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient(hunting): picked mon.noname-b con 0x7f76fc1457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc06ef30 con 0x7f76fc13f650 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc06f580 con 0x7f76fc1457f0 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76fc1422b0 con 0x7f76fc0a9d70 2024-03-29T11:27:22.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:22.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.966+0000 7f770190e640 10 monclient(hunting): authenticate will time out at 1363.229806s 2024-03-29T11:27:22.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.971+0000 7f76faffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f76fc0a9d70 0x7f76fc141ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46612/0 (socket says 172.21.15.19:46612) 2024-03-29T11:27:22.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.971+0000 7f76faffd640 1 -- 172.21.15.19:0/3417958018 learned_addr learned my addr 172.21.15.19:0/3417958018 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:22.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.972+0000 7f76f8ff9640 10 client.?.objecter ms_handle_connect 0x7f76fc0a9d70 2024-03-29T11:27:22.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.972+0000 7f76f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:22.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 client.?.objecter ms_handle_connect 0x7f76fc13f650 2024-03-29T11:27:22.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:22.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757322554 0 0) 0x7f76fc1422b0 con 0x7f76fc0a9d70 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc004eb0 con 0x7f76fc0a9d70 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138578806 0 0) 0x7f76fc06ef30 con 0x7f76fc13f650 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.973+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76fc1422b0 con 0x7f76fc13f650 2024-03-29T11:27:22.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3952093520 0 0) 0x7f76fc1422b0 con 0x7f76fc13f650 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76fc06ef30 con 0x7f76fc13f650 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075162848 0 0) 0x7f76dc004eb0 con 0x7f76fc0a9d70 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76fc1422b0 con 0x7f76fc0a9d70 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f76e40014b0 con 0x7f76fc13f650 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.971 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f76ec001750 con 0x7f76fc0a9d70 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.972 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2864649557 0 0) 0x7f76fc06ef30 con 0x7f76fc13f650 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 >> v1:172.21.15.19:6790/0 conn(0x7f76fc0a9d70 legacy=0x7f76fc141ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 >> v1:172.21.15.62:6789/0 conn(0x7f76fc1457f0 legacy=0x7f76fc1e6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 monclient: found mon.a 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:22.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76fc06f160 con 0x7f76fc13f650 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f76f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f770190e640 5 monclient: authenticate success, global_id 4506 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f770190e640 10 monclient: _renew_subs 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f770190e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.974+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76fc142540 con 0x7f76fc13f650 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.975+0000 7f770190e640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.975+0000 7f770190e640 10 monclient: _renew_subs 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.975+0000 7f770190e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.975+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76fc072050 con 0x7f76fc13f650 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.975+0000 7f770190e640 1 librados: init done 2024-03-29T11:27:22.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.978+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e4001610 con 0x7f76fc13f650 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.978+0000 7f76f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.978+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f76e40017c0 con 0x7f76fc13f650 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.978+0000 7f76f9ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.978+0000 7f770190e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f770190e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f770190e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e0005080 con 0x7f76fc13f650 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f76f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f76f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f76f8ff9640 10 monclient: dump: 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.975 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.979+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f76e401ac80 con 0x7f76fc13f650 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.980+0000 7f76faffd640 10 monclient: get_auth_request con 0x7f76dc06e030 auth_method 0 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.981+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f76e4084a00 con 0x7f76fc13f650 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.981+0000 7f76f8ff9640 10 client.4506.objecter ms_dispatch 0x7f76fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.981+0000 7f76f8ff9640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.981+0000 7f76f8ff9640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:22.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.981+0000 7f76f8ff9640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-29T11:27:22.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.981+0000 7f76f8ff9640 10 client.4506.objecter ms_handle_connect 0x7f76dc06e030 2024-03-29T11:27:22.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.989+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76e4050450 con 0x7f76fc13f650 2024-03-29T11:27:22.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.990+0000 7f76f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:22.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:22.990+0000 7f76f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:23.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.332+0000 7f770190e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:23.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.332+0000 7f770190e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:23.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.332+0000 7f770190e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:23.320 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.332+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76e0005330 con 0x7f76fc13f650 2024-03-29T11:27:23.326 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:23.326 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.335+0000 7f76f8ff9640 1 -- 172.21.15.19:0/3417958018 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f76e4052380 con 0x7f76fc13f650 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.335+0000 7f76f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.335+0000 7f76f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.338+0000 7f770190e640 10 librados: watch_flush enter 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 10 librados: watch_flush exit 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 10 client.4506.objecter successfully canceled tick 2024-03-29T11:27:23.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 >> v1:172.21.15.19:6816/3721232097 conn(0x7f76dc06e030 legacy=0x7f76dc0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 10 monclient: shutdown 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 >> v1:172.21.15.19:6789/0 conn(0x7f76fc13f650 legacy=0x7f76fc1e33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 shutdown_connections 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 >> 172.21.15.19:0/3417958018 conn(0x7f76fc066a00 msgr2=0x7f76fc13c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 shutdown_connections 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 -- 172.21.15.19:0/3417958018 wait complete. 2024-03-29T11:27:23.329 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:23.339+0000 7f770190e640 1 librados: shutdown 2024-03-29T11:27:23.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:30.152 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.317+0000 7ffbd0f54640 10 monclient: get_monmap_and_config 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.317+0000 7ffbd0f54640 10 monclient: build_initial_monmap 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.317+0000 7ffbd0f54640 10 monclient: monmap: 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:30.318590+0000 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:30.318590+0000 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:30.306 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:30.307 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:30.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.319+0000 7ffbd0f54640 1 Processor -- start 2024-03-29T11:27:30.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.319+0000 7ffbd0f54640 1 -- start start 2024-03-29T11:27:30.310 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.319+0000 7ffbd0f54640 10 monclient: init 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient(hunting): picked mon.noname-a con 0x7ffbcc149ed0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient(hunting): picked mon.noname-c con 0x7ffbcc072370 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient(hunting): picked mon.noname-b con 0x7ffbcc14a2b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc14aa50 con 0x7ffbcc149ed0 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc14a760 con 0x7ffbcc072370 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc070190 con 0x7ffbcc14a2b0 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:30.311 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.320+0000 7ffbd0f54640 10 monclient(hunting): authenticate will time out at 1370.583899s 2024-03-29T11:27:30.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.324+0000 7ffbca59c640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ffbcc149ed0 0x7ffbcc071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:54713/0 (socket says 172.21.15.19:54713) 2024-03-29T11:27:30.312 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.324+0000 7ffbca59c640 1 -- 172.21.15.19:0/2849124410 learned_addr learned my addr 172.21.15.19:0/2849124410 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:30.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538310597 0 0) 0x7ffbcc14aa50 con 0x7ffbcc149ed0 2024-03-29T11:27:30.313 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 10 monclient(hunting): _negotiate 2024-03-29T11:27:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:27:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb4003520 con 0x7ffbcc149ed0 2024-03-29T11:27:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3787547936 0 0) 0x7ffbb4003520 con 0x7ffbcc149ed0 2024-03-29T11:27:30.314 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.325+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 >> v1:172.21.15.19:6790/0 conn(0x7ffbcc14a2b0 legacy=0x7ffbcc06c860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 >> v1:172.21.15.62:6789/0 conn(0x7ffbcc072370 legacy=0x7ffbcc06c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: _finish_hunting 0 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 1 monclient: found mon.noname-a 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc070470 con 0x7ffbcc149ed0 2024-03-29T11:27:30.317 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: _finish_auth 0 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: _check_auth_tickets 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbd0f54640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbd0f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffbbc001690 con 0x7ffbcc149ed0 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: dump: 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:30.318 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbd0f54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbbc0018b0 con 0x7ffbcc149ed0 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 1 -- 172.21.15.19:0/2849124410 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffbbc001c40 con 0x7ffbcc149ed0 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:30.319 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.329+0000 7ffbc959a640 10 monclient: dump: 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:30.320 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:30.321 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.330+0000 7ffbd0f54640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.330+0000 7ffbd0f54640 4 set_mon_vals no callback set 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.331+0000 7ffbd0f54640 10 monclient: shutdown 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.331+0000 7ffbd0f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.331+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2849124410 >> v1:172.21.15.19:6789/0 conn(0x7ffbcc149ed0 legacy=0x7ffbcc071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.332+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2849124410 shutdown_connections 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.332+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2849124410 >> 172.21.15.19:0/2849124410 conn(0x7ffbcc0675b0 msgr2=0x7ffbcc0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.332+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2849124410 shutdown_connections 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.332+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2849124410 wait complete. 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.333+0000 7ffbd0f54640 10 monclient: build_initial_monmap 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.333+0000 7ffbd0f54640 10 monclient: monmap: 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:30.322 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.333+0000 7ffbd0f54640 1 librados: starting msgr at 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.333+0000 7ffbd0f54640 1 librados: starting objecter 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 1 Processor -- start 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 1 -- start start 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 1 librados: setting wanted keys 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 1 librados: calling monclient init 2024-03-29T11:27:30.323 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 10 monclient: init 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 10 monclient(hunting): picked mon.noname-c con 0x7ffbcc072370 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 10 monclient(hunting): picked mon.noname-b con 0x7ffbcc149ed0 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.334+0000 7ffbd0f54640 10 monclient(hunting): picked mon.noname-a con 0x7ffbcc14a2b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.335+0000 7ffbd0f54640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc1e3e50 con 0x7ffbcc14a2b0 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.335+0000 7ffbd0f54640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc0af950 con 0x7ffbcc149ed0 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.335+0000 7ffbd0f54640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc0ad5d0 con 0x7ffbcc072370 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.335+0000 7ffbd0f54640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:30.324 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.335+0000 7ffbd0f54640 10 monclient(hunting): authenticate will time out at 1370.598027s 2024-03-29T11:27:30.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.338+0000 7ffbcad9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ffbcc14a2b0 0x7ffbcc128a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:54809/0 (socket says 172.21.15.19:54809) 2024-03-29T11:27:30.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.338+0000 7ffbc9d9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7ffbcc149ed0 0x7ffbcc0ae040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:43217/0 (socket says 172.21.15.19:43217) 2024-03-29T11:27:30.326 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.338+0000 7ffbc9d9b640 1 -- 172.21.15.19:0/2078265949 learned_addr learned my addr 172.21.15.19:0/2078265949 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.338+0000 7ffbbaffd640 10 client.?.objecter ms_handle_connect 0x7ffbcc149ed0 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.338+0000 7ffbbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 client.?.objecter ms_handle_connect 0x7ffbcc14a2b0 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002568819 0 0) 0x7ffbcc0af950 con 0x7ffbcc149ed0 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:30.327 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbac004eb0 con 0x7ffbcc149ed0 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1320076901 0 0) 0x7ffbcc1e3e50 con 0x7ffbcc14a2b0 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:27:30.328 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc0af950 con 0x7ffbcc14a2b0 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1279372257 0 0) 0x7ffbac004eb0 con 0x7ffbcc149ed0 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbcc1e3e50 con 0x7ffbcc149ed0 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffbc0001930 con 0x7ffbcc149ed0 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.339+0000 7ffbbaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 10 monclient(hunting): dump: 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:30.332 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 978886093 0 0) 0x7ffbcc1e3e50 con 0x7ffbcc149ed0 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 >> v1:172.21.15.19:6790/0 conn(0x7ffbcc072370 legacy=0x7ffbcc1e4300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 >> v1:172.21.15.19:6789/0 conn(0x7ffbcc14a2b0 legacy=0x7ffbcc128a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 1 monclient: found mon.b 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.340+0000 7ffbbaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc0aaa40 con 0x7ffbcc149ed0 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:30.333 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 10 monclient: _renew_subs 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc0ae750 con 0x7ffbcc149ed0 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 10 monclient: _renew_subs 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc1e4bf0 con 0x7ffbcc149ed0 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbd0f54640 1 librados: init done 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbc00035f0 con 0x7ffbcc149ed0 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ffbc0003a00 con 0x7ffbcc149ed0 2024-03-29T11:27:30.334 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.341+0000 7ffbbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.342+0000 7ffbbaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.342+0000 7ffbc8d99640 4 set_mon_vals no callback set 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.342+0000 7ffbbaffd640 10 monclient: dump: 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:30.335 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.344+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ffbc0003b60 con 0x7ffbcc149ed0 2024-03-29T11:27:30.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ffbc0084a60 con 0x7ffbcc149ed0 2024-03-29T11:27:30.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbca59c640 10 monclient: get_auth_request con 0x7ffbac06dd00 auth_method 0 2024-03-29T11:27:30.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbbaffd640 10 client.4510.objecter ms_dispatch 0x7ffbcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:30.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbbaffd640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:30.337 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbbaffd640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:30.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbbaffd640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:27:30.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.345+0000 7ffbbaffd640 10 client.4510.objecter ms_handle_connect 0x7ffbac06dd00 2024-03-29T11:27:30.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.347+0000 7ffbd0f54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:30.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.347+0000 7ffbd0f54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:30.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.347+0000 7ffbd0f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:27:30.338 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.347+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0005080 con 0x7ffbcc149ed0 2024-03-29T11:27:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.360+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbc0050340 con 0x7ffbcc149ed0 2024-03-29T11:27:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.360+0000 7ffbbaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:30.348 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.360+0000 7ffbbaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.694+0000 7ffbd0f54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.694+0000 7ffbd0f54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.694+0000 7ffbd0f54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.694+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbb0005330 con 0x7ffbcc149ed0 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.695+0000 7ffbbaffd640 1 -- 172.21.15.19:0/2078265949 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ffbc0052270 con 0x7ffbcc149ed0 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.695+0000 7ffbbaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:30.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.695+0000 7ffbbaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:30.684 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:30.684 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:30.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 10 librados: watch_flush enter 2024-03-29T11:27:30.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 10 librados: watch_flush exit 2024-03-29T11:27:30.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:27:30.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:27:30.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 >> v1:172.21.15.19:6816/3721232097 conn(0x7ffbac06dd00 legacy=0x7ffbac0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 10 monclient: shutdown 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 >> v1:172.21.15.62:6789/0 conn(0x7ffbcc149ed0 legacy=0x7ffbcc0ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 shutdown_connections 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.699+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 >> 172.21.15.19:0/2078265949 conn(0x7ffbcc0675b0 msgr2=0x7ffbcc0b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.700+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 shutdown_connections 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.700+0000 7ffbd0f54640 1 -- 172.21.15.19:0/2078265949 wait complete. 2024-03-29T11:27:30.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:30.700+0000 7ffbd0f54640 1 librados: shutdown 2024-03-29T11:27:30.706 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:37.513 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.671+0000 7fae30515640 10 monclient: get_monmap_and_config 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.671+0000 7fae30515640 10 monclient: build_initial_monmap 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.671+0000 7fae30515640 10 monclient: monmap: 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:37.672679+0000 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:37.672679+0000 2024-03-29T11:27:37.660 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:37.661 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:37.661 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:37.661 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:37.661 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:37.661 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.672+0000 7fae30515640 1 Processor -- start 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.673+0000 7fae30515640 1 -- start start 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.673+0000 7fae30515640 10 monclient: init 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.674+0000 7fae30515640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.674+0000 7fae30515640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.674+0000 7fae30515640 10 monclient(hunting): picked mon.noname-c con 0x7fae280b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:37.662 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.674+0000 7fae30515640 10 monclient(hunting): picked mon.noname-b con 0x7fae2814a700 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.678+0000 7fae30515640 10 monclient(hunting): picked mon.noname-a con 0x7fae280ac5a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.678+0000 7fae30515640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280aba70 con 0x7fae280ac5a0 2024-03-29T11:27:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.678+0000 7fae30515640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280aecb0 con 0x7fae280b0770 2024-03-29T11:27:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.678+0000 7fae30515640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280af150 con 0x7fae2814a700 2024-03-29T11:27:37.666 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.678+0000 7fae30515640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:37.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.678+0000 7fae30515640 10 monclient(hunting): authenticate will time out at 1377.941238s 2024-03-29T11:27:37.667 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.679+0000 7fae2eab2640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fae280ac5a0 0x7fae28151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:53917/0 (socket says 172.21.15.19:53917) 2024-03-29T11:27:37.668 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.679+0000 7fae2eab2640 1 -- 172.21.15.19:0/1108367291 learned_addr learned my addr 172.21.15.19:0/1108367291 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948289230 0 0) 0x7fae280aba70 con 0x7fae280ac5a0 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 10 monclient(hunting): _negotiate 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae14003520 con 0x7fae280ac5a0 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.682+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792210118 0 0) 0x7fae280af150 con 0x7fae2814a700 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): _negotiate 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:37.674 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae280aba70 con 0x7fae2814a700 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253760351 0 0) 0x7fae280aecb0 con 0x7fae280b0770 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): _negotiate 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae280af150 con 0x7fae280b0770 2024-03-29T11:27:37.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 790133162 0 0) 0x7fae280af150 con 0x7fae280b0770 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 >> v1:172.21.15.19:6790/0 conn(0x7fae2814a700 legacy=0x7fae280abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 >> v1:172.21.15.19:6789/0 conn(0x7fae280ac5a0 legacy=0x7fae28151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient: _finish_hunting 0 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 1 monclient: found mon.noname-c 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.683+0000 7fae2d2af640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280ac9d0 con 0x7fae280b0770 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 10 monclient: _finish_auth 0 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 10 monclient: _check_auth_tickets 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae30515640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae30515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:37.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae24001630 con 0x7fae280b0770 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 10 monclient: dump: 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:37.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae30515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 1 -- 172.21.15.19:0/1108367291 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae24001790 con 0x7fae280b0770 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae2d2af640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae30515640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.684+0000 7fae30515640 4 set_mon_vals no callback set 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.685+0000 7fae30515640 10 monclient: shutdown 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.685+0000 7fae30515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.685+0000 7fae30515640 1 -- 172.21.15.19:0/1108367291 >> v1:172.21.15.62:6789/0 conn(0x7fae280b0770 legacy=0x7fae28149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.686+0000 7fae30515640 1 -- 172.21.15.19:0/1108367291 shutdown_connections 2024-03-29T11:27:37.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.686+0000 7fae30515640 1 -- 172.21.15.19:0/1108367291 >> 172.21.15.19:0/1108367291 conn(0x7fae2806b660 msgr2=0x7fae2806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.687+0000 7fae30515640 1 -- 172.21.15.19:0/1108367291 shutdown_connections 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.688+0000 7fae30515640 1 -- 172.21.15.19:0/1108367291 wait complete. 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.689+0000 7fae30515640 10 monclient: build_initial_monmap 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.689+0000 7fae30515640 10 monclient: monmap: 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:37.680 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.689+0000 7fae30515640 1 librados: starting msgr at 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.689+0000 7fae30515640 1 librados: starting objecter 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.690+0000 7fae30515640 1 Processor -- start 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.690+0000 7fae30515640 1 -- start start 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.690+0000 7fae30515640 1 librados: setting wanted keys 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.690+0000 7fae30515640 1 librados: calling monclient init 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.690+0000 7fae30515640 10 monclient: init 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.691+0000 7fae30515640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:37.681 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.691+0000 7fae30515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:37.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.691+0000 7fae30515640 10 monclient(hunting): picked mon.noname-c con 0x7fae280ac5a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:37.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.694+0000 7fae30515640 10 monclient(hunting): picked mon.noname-a con 0x7fae280b0770 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:37.684 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.697+0000 7fae2e2b1640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fae280ac5a0 0x7fae281df250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:48727/0 (socket says 172.21.15.19:48727) 2024-03-29T11:27:37.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.697+0000 7fae30515640 10 monclient(hunting): picked mon.noname-b con 0x7fae2814a700 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:37.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.697+0000 7fae30515640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae281df960 con 0x7fae280b0770 2024-03-29T11:27:37.686 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae30515640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae281dfc00 con 0x7fae2814a700 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae30515640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae281459f0 con 0x7fae280ac5a0 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae30515640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae30515640 10 monclient(hunting): authenticate will time out at 1377.961012s 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae2e2b1640 1 -- 172.21.15.19:0/1583240700 learned_addr learned my addr 172.21.15.19:0/1583240700 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 client.?.objecter ms_handle_connect 0x7fae280ac5a0 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364754726 0 0) 0x7fae281459f0 con 0x7fae280ac5a0 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae0c004eb0 con 0x7fae280ac5a0 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2815124609 0 0) 0x7fae0c004eb0 con 0x7fae280ac5a0 2024-03-29T11:27:37.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae281459f0 con 0x7fae280ac5a0 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae24003890 con 0x7fae280ac5a0 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): dump: 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:37.688 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:37.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.698+0000 7fae1effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.699+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2946618135 0 0) 0x7fae281459f0 con 0x7fae280ac5a0 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.699+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 >> v1:172.21.15.62:6789/0 conn(0x7fae2814a700 legacy=0x7fae281261e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.699+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 >> v1:172.21.15.19:6789/0 conn(0x7fae280b0770 legacy=0x7fae281443d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.700+0000 7fae1effd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 1 monclient: found mon.c 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae28144ae0 con 0x7fae280ac5a0 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 10 monclient: _finish_auth 0 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:37.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.701+0000 7fae1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:37.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.702+0000 7fae30515640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:27:37.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.702+0000 7fae30515640 10 monclient: _renew_subs 2024-03-29T11:27:37.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.702+0000 7fae30515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:37.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.702+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae28144d90 con 0x7fae280ac5a0 2024-03-29T11:27:37.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.702+0000 7fae30515640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.702+0000 7fae30515640 10 monclient: _renew_subs 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae30515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae24003e80 con 0x7fae280ac5a0 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae28145130 con 0x7fae280ac5a0 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae30515640 1 librados: init done 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fae240042d0 con 0x7fae280ac5a0 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae1effd640 10 monclient: dump: 2024-03-29T11:27:37.694 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.703+0000 7fae2caae640 4 set_mon_vals no callback set 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.705+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fae24001a00 con 0x7fae280ac5a0 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.705+0000 7fae30515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:37.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.705+0000 7fae30515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:37.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.705+0000 7fae30515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:37.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.705+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae280ab560 con 0x7fae280ac5a0 2024-03-29T11:27:37.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fae24085c60 con 0x7fae280ac5a0 2024-03-29T11:27:37.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae1effd640 10 client.4547.objecter ms_dispatch 0x7fae28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:37.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae1effd640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:37.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae1effd640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:37.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae2dab0640 10 monclient: get_auth_request con 0x7fae0c06dbc0 auth_method 0 2024-03-29T11:27:37.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae1effd640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:27:37.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.706+0000 7fae1effd640 10 client.4547.objecter ms_handle_connect 0x7fae0c06dbc0 2024-03-29T11:27:37.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.716+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae24050510 con 0x7fae280ac5a0 2024-03-29T11:27:37.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.716+0000 7fae1effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:37.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:37.716+0000 7fae1effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:38.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.037+0000 7fae30515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:38.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.037+0000 7fae30515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:38.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.037+0000 7fae30515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:38.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.037+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae28075f10 con 0x7fae280ac5a0 2024-03-29T11:27:38.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.040+0000 7fae1effd640 1 -- 172.21.15.19:0/1583240700 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fae24052440 con 0x7fae280ac5a0 2024-03-29T11:27:38.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.040+0000 7fae1effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:38.028 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.040+0000 7fae1effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:38.030 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:38.030 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:38.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.045+0000 7fae30515640 10 librados: watch_flush enter 2024-03-29T11:27:38.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.047+0000 7fae30515640 10 librados: watch_flush exit 2024-03-29T11:27:38.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.047+0000 7fae30515640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:27:38.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.047+0000 7fae30515640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:27:38.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.048+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 >> v1:172.21.15.19:6816/3721232097 conn(0x7fae0c06dbc0 legacy=0x7fae0c070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:38.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.048+0000 7fae30515640 10 monclient: shutdown 2024-03-29T11:27:38.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.048+0000 7fae30515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:38.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.048+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 >> v1:172.21.15.19:6790/0 conn(0x7fae280ac5a0 legacy=0x7fae281df250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:38.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.049+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 shutdown_connections 2024-03-29T11:27:38.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.049+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 >> 172.21.15.19:0/1583240700 conn(0x7fae2806b660 msgr2=0x7fae280b1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:38.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.049+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 shutdown_connections 2024-03-29T11:27:38.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.049+0000 7fae30515640 1 -- 172.21.15.19:0/1583240700 wait complete. 2024-03-29T11:27:38.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:38.049+0000 7fae30515640 1 librados: shutdown 2024-03-29T11:27:38.053 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:44.861 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:45.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.030+0000 7fbe63fff640 10 monclient: get_monmap_and_config 2024-03-29T11:27:45.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.030+0000 7fbe63fff640 10 monclient: build_initial_monmap 2024-03-29T11:27:45.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.030+0000 7fbe63fff640 10 monclient: monmap: 2024-03-29T11:27:45.019 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:45.030990+0000 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:45.030990+0000 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.031+0000 7fbe63fff640 1 Processor -- start 2024-03-29T11:27:45.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 1 -- start start 2024-03-29T11:27:45.021 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient: init 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5c034850 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5c0c9550 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c02aac0 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c02c440 con 0x7fbe5c034850 2024-03-29T11:27:45.022 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c02aed0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c02b190 con 0x7fbe5c0c9550 2024-03-29T11:27:45.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:45.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.032+0000 7fbe63fff640 10 monclient(hunting): authenticate will time out at 1385.295934s 2024-03-29T11:27:45.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.037+0000 7fbe637fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fbe5c02aac0 0x7fbe5c0cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:49337/0 (socket says 172.21.15.19:49337) 2024-03-29T11:27:45.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.037+0000 7fbe637fe640 1 -- 172.21.15.19:0/1818392057 learned_addr learned my addr 172.21.15.19:0/1818392057 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.038+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365751739 0 0) 0x7fbe5c02aed0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.038+0000 7fbe61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.038+0000 7fbe61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.038+0000 7fbe61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.038+0000 7fbe61ffb640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.038+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c003520 con 0x7fbe5c02aac0 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716635188 0 0) 0x7fbe5c02b190 con 0x7fbe5c0c9550 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:27:45.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c003840 con 0x7fbe5c0c9550 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571363368 0 0) 0x7fbe4c003520 con 0x7fbe5c02aac0 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 >> v1:172.21.15.19:6790/0 conn(0x7fbe5c0c9550 legacy=0x7fbe5c0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 >> v1:172.21.15.19:6789/0 conn(0x7fbe5c034850 legacy=0x7fbe5c0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 monclient: found mon.noname-c 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c033060 con 0x7fbe5c02aac0 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbe6404f730 con 0x7fbe5c02aac0 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:45.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: dump: 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe640008d0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 1 -- 172.21.15.19:0/1818392057 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbe6404f730 con 0x7fbe5c02aac0 2024-03-29T11:27:45.032 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.039+0000 7fbe61ffb640 10 monclient: dump: 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:45.033 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.041+0000 7fbe63fff640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.041+0000 7fbe63fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.041+0000 7fbe63fff640 4 set_mon_vals no callback set 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.042+0000 7fbe63fff640 10 monclient: shutdown 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.042+0000 7fbe63fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.042+0000 7fbe63fff640 1 -- 172.21.15.19:0/1818392057 >> v1:172.21.15.62:6789/0 conn(0x7fbe5c02aac0 legacy=0x7fbe5c0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.043+0000 7fbe63fff640 1 -- 172.21.15.19:0/1818392057 shutdown_connections 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.043+0000 7fbe63fff640 1 -- 172.21.15.19:0/1818392057 >> 172.21.15.19:0/1818392057 conn(0x7fbe5c0058f0 msgr2=0x7fbe5c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.043+0000 7fbe63fff640 1 -- 172.21.15.19:0/1818392057 shutdown_connections 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.043+0000 7fbe63fff640 1 -- 172.21.15.19:0/1818392057 wait complete. 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.044+0000 7fbe63fff640 10 monclient: build_initial_monmap 2024-03-29T11:27:45.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.044+0000 7fbe63fff640 10 monclient: monmap: 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.044+0000 7fbe63fff640 1 librados: starting msgr at 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.044+0000 7fbe63fff640 1 librados: starting objecter 2024-03-29T11:27:45.036 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.044+0000 7fbe63fff640 1 Processor -- start 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.045+0000 7fbe63fff640 1 -- start start 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.045+0000 7fbe63fff640 1 librados: setting wanted keys 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.045+0000 7fbe63fff640 1 librados: calling monclient init 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.045+0000 7fbe63fff640 10 monclient: init 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5c02aac0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5c034850 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:45.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c0c9550 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:45.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c032b20 con 0x7fbe5c02aac0 2024-03-29T11:27:45.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0c0300 con 0x7fbe5c034850 2024-03-29T11:27:45.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0c0f10 con 0x7fbe5c0c9550 2024-03-29T11:27:45.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:45.038 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.046+0000 7fbe63fff640 10 monclient(hunting): authenticate will time out at 1385.309741s 2024-03-29T11:27:45.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.050+0000 7fbe62ffd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fbe5c02aac0 0x7fbe5c0bfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:45633/0 (socket says 172.21.15.19:45633) 2024-03-29T11:27:45.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.050+0000 7fbe62ffd640 1 -- 172.21.15.19:0/3902271931 learned_addr learned my addr 172.21.15.19:0/3902271931 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 client.?.objecter ms_handle_connect 0x7fbe5c02aac0 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643052930 0 0) 0x7fbe5c032b20 con 0x7fbe5c02aac0 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe50004eb0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3816265568 0 0) 0x7fbe50004eb0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe50005300 con 0x7fbe5c02aac0 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 client.?.objecter ms_handle_connect 0x7fbe5c0c9550 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.051+0000 7fbe4bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbe54004040 con 0x7fbe5c02aac0 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): dump: 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:45.043 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555006219 0 0) 0x7fbe5c0c0f10 con 0x7fbe5c0c9550 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe500058a0 con 0x7fbe5c0c9550 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 92253984 0 0) 0x7fbe500058a0 con 0x7fbe5c0c9550 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.052+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe50005d30 con 0x7fbe5c0c9550 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbe6404f730 con 0x7fbe5c0c9550 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:45.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient(hunting): dump: 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3706232466 0 0) 0x7fbe50005300 con 0x7fbe5c02aac0 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 >> v1:172.21.15.19:6790/0 conn(0x7fbe5c0c9550 legacy=0x7fbe5c0bc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:45.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 >> v1:172.21.15.62:6789/0 conn(0x7fbe5c034850 legacy=0x7fbe5c0bc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 monclient: found mon.a 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c0c11f0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: _finish_auth 0 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 10 monclient: _renew_subs 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5c0bee20 con 0x7fbe5c02aac0 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:45.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 10 monclient: _renew_subs 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5c175050 con 0x7fbe5c02aac0 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe63fff640 1 librados: init done 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe54000ed0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fbe54004040 con 0x7fbe5c02aac0 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe4bfff640 10 monclient: dump: 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:45.047 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:45.048 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:45.048 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:45.048 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:45.048 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:45.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.053+0000 7fbe617fa640 4 set_mon_vals no callback set 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.054+0000 7fbe63fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.054+0000 7fbe63fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.054+0000 7fbe63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.054+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5c0bf010 con 0x7fbe5c02aac0 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.055+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fbe54001180 con 0x7fbe5c02aac0 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.056+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fbe540543f0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.056+0000 7fbe4bfff640 10 client.4518.objecter ms_dispatch 0x7fbe64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.056+0000 7fbe4bfff640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:45.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.056+0000 7fbe4bfff640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:45.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.056+0000 7fbe4bfff640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:27:45.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.058+0000 7fbe627fc640 10 monclient: get_auth_request con 0x7fbe5006e8c0 auth_method 0 2024-03-29T11:27:45.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.059+0000 7fbe4bfff640 10 client.4518.objecter ms_handle_connect 0x7fbe5006e8c0 2024-03-29T11:27:45.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.064+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe54050690 con 0x7fbe5c02aac0 2024-03-29T11:27:45.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.064+0000 7fbe4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:45.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.064+0000 7fbe4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:45.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.401+0000 7fbe63fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:45.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.401+0000 7fbe63fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:45.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.401+0000 7fbe63fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:27:45.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.401+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe5c004cd0 con 0x7fbe5c02aac0 2024-03-29T11:27:45.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.403+0000 7fbe4bfff640 1 -- 172.21.15.19:0/3902271931 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fbe54050690 con 0x7fbe5c02aac0 2024-03-29T11:27:45.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.403+0000 7fbe4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:45.392 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.403+0000 7fbe4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:45.393 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:45.393 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:45.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.408+0000 7fbe63fff640 10 librados: watch_flush enter 2024-03-29T11:27:45.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.408+0000 7fbe63fff640 10 librados: watch_flush exit 2024-03-29T11:27:45.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.408+0000 7fbe63fff640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:27:45.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.408+0000 7fbe63fff640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:27:45.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.408+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 >> v1:172.21.15.19:6816/3721232097 conn(0x7fbe5006e8c0 legacy=0x7fbe50070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:45.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 10 monclient: shutdown 2024-03-29T11:27:45.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:45.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 >> v1:172.21.15.19:6789/0 conn(0x7fbe5c02aac0 legacy=0x7fbe5c0bfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:45.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 shutdown_connections 2024-03-29T11:27:45.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 >> 172.21.15.19:0/3902271931 conn(0x7fbe5c0058f0 msgr2=0x7fbe5c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:45.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 shutdown_connections 2024-03-29T11:27:45.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 1 -- 172.21.15.19:0/3902271931 wait complete. 2024-03-29T11:27:45.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:45.409+0000 7fbe63fff640 1 librados: shutdown 2024-03-29T11:27:45.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:52.225 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:52.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.385+0000 7f8544510640 10 monclient: get_monmap_and_config 2024-03-29T11:27:52.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.385+0000 7f8544510640 10 monclient: build_initial_monmap 2024-03-29T11:27:52.375 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.385+0000 7f8544510640 10 monclient: monmap: 2024-03-29T11:27:52.375 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:52.375 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:52.375 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:52.386637+0000 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:52.386637+0000 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:52.376 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.386+0000 7f8544510640 1 Processor -- start 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.386+0000 7f8544510640 1 -- start start 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.386+0000 7f8544510640 10 monclient: init 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient(hunting): picked mon.noname-a con 0x7f853c1422b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient(hunting): picked mon.noname-b con 0x7f853c142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient(hunting): picked mon.noname-c con 0x7f853c0b17d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c142c80 con 0x7f853c1422b0 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0aae40 con 0x7f853c0b17d0 2024-03-29T11:27:52.381 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c13fbd0 con 0x7f853c142690 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.387+0000 7f8544510640 10 monclient(hunting): authenticate will time out at 1392.650901s 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f8541aab640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f853c142690 0x7f853c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:42588/0 (socket says 172.21.15.19:42588) 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f8541aab640 1 -- 172.21.15.19:0/1081301736 learned_addr learned my addr 172.21.15.19:0/1081301736 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539698123 0 0) 0x7f853c13fbd0 con 0x7f853c142690 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f85412aa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f85412aa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f85412aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f85412aa640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:27:52.382 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.392+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8528003520 con 0x7f853c142690 2024-03-29T11:27:52.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 303783055 0 0) 0x7f8528003520 con 0x7f853c142690 2024-03-29T11:27:52.384 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 >> v1:172.21.15.62:6789/0 conn(0x7f853c0b17d0 legacy=0x7f853c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 >> v1:172.21.15.19:6789/0 conn(0x7f853c1422b0 legacy=0x7f853c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 monclient: found mon.noname-b 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c151940 con 0x7f853c142690 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: _finish_auth 0 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:52.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8538001690 con 0x7f853c142690 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: dump: 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:52.386 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85380018b0 con 0x7f853c142690 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 1 -- 172.21.15.19:0/1081301736 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8538001c40 con 0x7f853c142690 2024-03-29T11:27:52.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:52.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:52.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f85412aa640 10 monclient: dump: 2024-03-29T11:27:52.388 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f8544510640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f8544510640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.393+0000 7f8544510640 4 set_mon_vals no callback set 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.394+0000 7f8544510640 10 monclient: shutdown 2024-03-29T11:27:52.389 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.394+0000 7f8544510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:52.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.394+0000 7f8544510640 1 -- 172.21.15.19:0/1081301736 >> v1:172.21.15.19:6790/0 conn(0x7f853c142690 legacy=0x7f853c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:52.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.394+0000 7f8544510640 1 -- 172.21.15.19:0/1081301736 shutdown_connections 2024-03-29T11:27:52.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.394+0000 7f8544510640 1 -- 172.21.15.19:0/1081301736 >> 172.21.15.19:0/1081301736 conn(0x7f853c066a00 msgr2=0x7f853c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:52.390 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.395+0000 7f8544510640 1 -- 172.21.15.19:0/1081301736 shutdown_connections 2024-03-29T11:27:52.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.395+0000 7f8544510640 1 -- 172.21.15.19:0/1081301736 wait complete. 2024-03-29T11:27:52.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 10 monclient: build_initial_monmap 2024-03-29T11:27:52.395 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 10 monclient: monmap: 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 1 librados: starting msgr at 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 1 librados: starting objecter 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 1 Processor -- start 2024-03-29T11:27:52.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 1 -- start start 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 1 librados: setting wanted keys 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 1 librados: calling monclient init 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.396+0000 7f8544510640 10 monclient: init 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient(hunting): picked mon.noname-a con 0x7f853c0b17d0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient(hunting): picked mon.noname-c con 0x7f853c1422b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient(hunting): picked mon.noname-b con 0x7f853c142690 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c12f680 con 0x7f853c0b17d0 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c06e980 con 0x7f853c142690 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c06ec60 con 0x7f853c1422b0 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.397+0000 7f8544510640 10 monclient(hunting): authenticate will time out at 1392.660647s 2024-03-29T11:27:52.397 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.398+0000 7f8541aab640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f853c1422b0 0x7f853c12e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:44240/0 (socket says 172.21.15.19:44240) 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.398+0000 7f8541aab640 1 -- 172.21.15.19:0/1856979450 learned_addr learned my addr 172.21.15.19:0/1856979450 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 10 client.?.objecter ms_handle_connect 0x7f853c1422b0 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640081645 0 0) 0x7f853c06ec60 con 0x7f853c1422b0 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:52.398 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.399+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f853c1422b0 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927374178 0 0) 0x7f8520004eb0 con 0x7f853c1422b0 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853c06ec60 con 0x7f853c1422b0 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f85380015d0 con 0x7f853c1422b0 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 10 monclient(hunting): dump: 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:52.399 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2489770849 0 0) 0x7f853c06ec60 con 0x7f853c1422b0 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.400+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 >> v1:172.21.15.62:6789/0 conn(0x7f853c142690 legacy=0x7f853c12ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 >> v1:172.21.15.19:6789/0 conn(0x7f853c0b17d0 legacy=0x7f853c150ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 1 monclient: found mon.c 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c06ef40 con 0x7f853c1422b0 2024-03-29T11:27:52.400 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 10 monclient: _renew_subs 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c150540 con 0x7f853c1422b0 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 10 monclient: _renew_subs 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c1500b0 con 0x7f853c1422b0 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8544510640 1 librados: init done 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 10 client.4559.objecter ms_handle_connect 0x7f853c0b17d0 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.404+0000 7f8532ffd640 10 client.4559.objecter resend_mon_ops 2024-03-29T11:27:52.401 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8544510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8544510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8544510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c150360 con 0x7f853c1422b0 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8538001730 con 0x7f853c1422b0 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8532ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f8538004c30 con 0x7f853c1422b0 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8532ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8532ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8532ffd640 10 monclient: dump: 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:52.402 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:52.403 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:52.403 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:52.403 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:52.403 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:52.403 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:52.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.405+0000 7f8533fff640 4 set_mon_vals no callback set 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.407+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f853801ac00 con 0x7f853c1422b0 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.408+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f8538084b60 con 0x7f853c1422b0 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.408+0000 7f8532ffd640 10 client.4559.objecter ms_dispatch 0x7f853c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.408+0000 7f8532ffd640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.408+0000 7f8532ffd640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.408+0000 7f8532ffd640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.408+0000 7f85422ac640 10 monclient: get_auth_request con 0x7f852006dbc0 auth_method 0 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.409+0000 7f8532ffd640 10 client.4559.objecter ms_handle_connect 0x7f852006dbc0 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.415+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8538050440 con 0x7f853c1422b0 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.415+0000 7f8532ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:52.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.415+0000 7f8532ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:52.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.736+0000 7f8544510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:52.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.736+0000 7f8544510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:52.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.736+0000 7f8544510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:52.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.736+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f853c075f10 con 0x7f853c1422b0 2024-03-29T11:27:52.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.737+0000 7f8532ffd640 1 -- 172.21.15.19:0/1856979450 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f8538052370 con 0x7f853c1422b0 2024-03-29T11:27:52.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.737+0000 7f8532ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:52.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.737+0000 7f8532ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:52.727 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:27:52.727 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:52.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.740+0000 7f8544510640 10 librados: watch_flush enter 2024-03-29T11:27:52.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 10 librados: watch_flush exit 2024-03-29T11:27:52.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 >> v1:172.21.15.19:6816/3721232097 conn(0x7f852006dbc0 legacy=0x7f8520070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 10 monclient: shutdown 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 >> v1:172.21.15.19:6790/0 conn(0x7f853c1422b0 legacy=0x7f853c12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 shutdown_connections 2024-03-29T11:27:52.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.741+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 >> 172.21.15.19:0/1856979450 conn(0x7f853c066a00 msgr2=0x7f853c0b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:52.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.742+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 shutdown_connections 2024-03-29T11:27:52.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.742+0000 7f8544510640 1 -- 172.21.15.19:0/1856979450 wait complete. 2024-03-29T11:27:52.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:52.742+0000 7f8544510640 1 librados: shutdown 2024-03-29T11:27:52.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:59.554 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:59.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.727+0000 7f66bb9c4640 10 monclient: get_monmap_and_config 2024-03-29T11:27:59.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.727+0000 7f66bb9c4640 10 monclient: build_initial_monmap 2024-03-29T11:27:59.717 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.727+0000 7f66bb9c4640 10 monclient: monmap: 2024-03-29T11:27:59.717 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:59.717 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:27:59.729190+0000 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:27:59.729190+0000 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:27:59.718 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:59.719 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.728+0000 7f66bb9c4640 1 Processor -- start 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.728+0000 7f66bb9c4640 1 -- start start 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.728+0000 7f66bb9c4640 10 monclient: init 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f66b40708a0 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient(hunting): picked mon.noname-c con 0x7f66b406fe90 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f66b4143c50 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b4072630 con 0x7f66b40708a0 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b4071410 con 0x7f66b406fe90 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b40afaa0 con 0x7f66b4143c50 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:59.720 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66bb9c4640 10 monclient(hunting): authenticate will time out at 1399.993652s 2024-03-29T11:27:59.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.729+0000 7f66b8f5f640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f66b406fe90 0x7f66b4143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:37908/0 (socket says 172.21.15.19:37908) 2024-03-29T11:27:59.721 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b8f5f640 1 -- 172.21.15.19:0/3567348873 learned_addr learned my addr 172.21.15.19:0/3567348873 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521510518 0 0) 0x7f66b4071410 con 0x7f66b406fe90 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b3fff640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.730+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66a4003520 con 0x7f66b406fe90 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1787098812 0 0) 0x7f66a4003520 con 0x7f66b406fe90 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 >> v1:172.21.15.19:6790/0 conn(0x7f66b4143c50 legacy=0x7f66b414b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 >> v1:172.21.15.19:6789/0 conn(0x7f66b40708a0 legacy=0x7f66b40af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.722 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 1 monclient: found mon.noname-c 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b40ac8b0 con 0x7f66b406fe90 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: _finish_auth 0 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66bb9c4640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66bb9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f66a0001690 con 0x7f66b406fe90 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66b3fff640 10 monclient: dump: 2024-03-29T11:27:59.723 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:59.724 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.731+0000 7f66bb9c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:59.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66b3fff640 1 -- 172.21.15.19:0/3567348873 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66a0001870 con 0x7f66b406fe90 2024-03-29T11:27:59.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:59.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66bb9c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:59.725 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66bb9c4640 4 set_mon_vals no callback set 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66bb9c4640 10 monclient: shutdown 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66bb9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.733+0000 7f66bb9c4640 1 -- 172.21.15.19:0/3567348873 >> v1:172.21.15.62:6789/0 conn(0x7f66b406fe90 legacy=0x7f66b4143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.734+0000 7f66bb9c4640 1 -- 172.21.15.19:0/3567348873 shutdown_connections 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.734+0000 7f66bb9c4640 1 -- 172.21.15.19:0/3567348873 >> 172.21.15.19:0/3567348873 conn(0x7f66b40675b0 msgr2=0x7f66b40b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.734+0000 7f66bb9c4640 1 -- 172.21.15.19:0/3567348873 shutdown_connections 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.734+0000 7f66bb9c4640 1 -- 172.21.15.19:0/3567348873 wait complete. 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.735+0000 7f66bb9c4640 10 monclient: build_initial_monmap 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.735+0000 7f66bb9c4640 10 monclient: monmap: 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:27:59.726 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.735+0000 7f66bb9c4640 1 librados: starting msgr at 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.735+0000 7f66bb9c4640 1 librados: starting objecter 2024-03-29T11:27:59.727 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.735+0000 7f66bb9c4640 1 Processor -- start 2024-03-29T11:27:59.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.736+0000 7f66bb9c4640 1 -- start start 2024-03-29T11:27:59.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.736+0000 7f66bb9c4640 1 librados: setting wanted keys 2024-03-29T11:27:59.732 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.736+0000 7f66bb9c4640 1 librados: calling monclient init 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.736+0000 7f66bb9c4640 10 monclient: init 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.736+0000 7f66bb9c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 10 monclient(hunting): picked mon.noname-a con 0x7f66b406fe90 addr v1:172.21.15.19:6789/0 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 10 monclient(hunting): picked mon.noname-b con 0x7f66b40708a0 addr v1:172.21.15.62:6789/0 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 10 monclient(hunting): picked mon.noname-c con 0x7f66b4143c50 addr v1:172.21.15.19:6790/0 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b413e920 con 0x7f66b406fe90 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b413ec00 con 0x7f66b40708a0 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b406d380 con 0x7f66b4143c50 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66bb9c4640 10 monclient(hunting): authenticate will time out at 1400.001087s 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b9f61640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f66b4143c50 0x7f66b41e65b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52168/0 (socket says 172.21.15.19:52168) 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b9f61640 1 -- 172.21.15.19:0/1165723671 learned_addr learned my addr 172.21.15.19:0/1165723671 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:27:59.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 client.?.objecter ms_handle_connect 0x7f66b4143c50 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035107868 0 0) 0x7f66b406d380 con 0x7f66b4143c50 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6698004eb0 con 0x7f66b4143c50 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 client.?.objecter ms_handle_connect 0x7f66b40708a0 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.737+0000 7f66b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 client.?.objecter ms_handle_connect 0x7f66b406fe90 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771394063 0 0) 0x7f66b413ec00 con 0x7f66b40708a0 2024-03-29T11:27:59.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b406d380 con 0x7f66b40708a0 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612848256 0 0) 0x7f6698004eb0 con 0x7f66b4143c50 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b413ec00 con 0x7f66b4143c50 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125043129 0 0) 0x7f66b413e920 con 0x7f66b406fe90 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6698004eb0 con 0x7f66b406fe90 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f66ac001930 con 0x7f66b4143c50 2024-03-29T11:27:59.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.738+0000 7f66b1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:59.736 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763031379 0 0) 0x7f66b406d380 con 0x7f66b40708a0 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f66b413e920 con 0x7f66b40708a0 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f66a0001d40 con 0x7f66b40708a0 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient(hunting): dump: 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:59.737 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:59.738 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:59.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746106435 0 0) 0x7f66b413ec00 con 0x7f66b4143c50 2024-03-29T11:27:59.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 >> v1:172.21.15.62:6789/0 conn(0x7f66b40708a0 legacy=0x7f66b4142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 >> v1:172.21.15.19:6789/0 conn(0x7f66b406fe90 legacy=0x7f66b4141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:59.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:59.738 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 monclient: found mon.c 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b413f090 con 0x7f66b4143c50 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 10 monclient: _renew_subs 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b406cf30 con 0x7f66b4143c50 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 10 monclient: _renew_subs 2024-03-29T11:27:59.739 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b4143000 con 0x7f66b4143c50 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66bb9c4640 1 librados: init done 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f66ac001a90 con 0x7f66b4143c50 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f66ac003870 con 0x7f66b4143c50 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b1ffb640 10 monclient: dump: 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:27:59.740 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.739+0000 7f66b37fe640 4 set_mon_vals no callback set 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.741+0000 7f66bb9c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.742+0000 7f66bb9c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.742+0000 7f66bb9c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:27:59.741 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.742+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6694005080 con 0x7f66b4143c50 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.743+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f66ac003ab0 con 0x7f66b4143c50 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.743+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f66ac084a60 con 0x7f66b4143c50 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.743+0000 7f66b1ffb640 10 client.4565.objecter ms_dispatch 0x7f66b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.743+0000 7f66b1ffb640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.743+0000 7f66b1ffb640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.743+0000 7f66b1ffb640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.744+0000 7f66b9760640 10 monclient: get_auth_request con 0x7f669806df40 auth_method 0 2024-03-29T11:27:59.742 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.744+0000 7f66b1ffb640 10 client.4565.objecter ms_handle_connect 0x7f669806df40 2024-03-29T11:27:59.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.753+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f66ac050340 con 0x7f66b4143c50 2024-03-29T11:27:59.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.753+0000 7f66b1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:59.743 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:27:59.753+0000 7f66b1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:00.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.107+0000 7f66bb9c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:00.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.107+0000 7f66bb9c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:00.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.107+0000 7f66bb9c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:00.098 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.107+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6694005330 con 0x7f66b4143c50 2024-03-29T11:28:00.099 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.110+0000 7f66b1ffb640 1 -- 172.21.15.19:0/1165723671 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f66ac052270 con 0x7f66b4143c50 2024-03-29T11:28:00.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.110+0000 7f66b1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:00.100 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.110+0000 7f66b1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:00.100 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:00.100 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 10 librados: watch_flush enter 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 10 librados: watch_flush exit 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 >> v1:172.21.15.19:6816/3721232097 conn(0x7f669806df40 legacy=0x7f6698070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 10 monclient: shutdown 2024-03-29T11:28:00.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:00.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.116+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 >> v1:172.21.15.19:6790/0 conn(0x7f66b4143c50 legacy=0x7f66b41e65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:00.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.117+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 shutdown_connections 2024-03-29T11:28:00.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.117+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 >> 172.21.15.19:0/1165723671 conn(0x7f66b40675b0 msgr2=0x7f66b413d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:00.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.117+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 shutdown_connections 2024-03-29T11:28:00.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.118+0000 7f66bb9c4640 1 -- 172.21.15.19:0/1165723671 wait complete. 2024-03-29T11:28:00.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:00.120+0000 7f66bb9c4640 1 librados: shutdown 2024-03-29T11:28:00.125 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:06.940 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:07.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.094+0000 7f4c2720f640 10 monclient: get_monmap_and_config 2024-03-29T11:28:07.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.094+0000 7f4c2720f640 10 monclient: build_initial_monmap 2024-03-29T11:28:07.084 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.094+0000 7f4c2720f640 10 monclient: monmap: 2024-03-29T11:28:07.084 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:07.096196+0000 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:07.096196+0000 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:07.085 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:07.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.095+0000 7f4c2720f640 1 Processor -- start 2024-03-29T11:28:07.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.095+0000 7f4c2720f640 1 -- start start 2024-03-29T11:28:07.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.095+0000 7f4c2720f640 10 monclient: init 2024-03-29T11:28:07.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:07.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:07.086 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient(hunting): picked mon.noname-a con 0x7f4c201422b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient(hunting): picked mon.noname-b con 0x7f4c20142690 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient(hunting): picked mon.noname-c con 0x7f4c200b17d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c20142c80 con 0x7f4c201422b0 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200aae40 con 0x7f4c200b17d0 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2013fbd0 con 0x7f4c20142690 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c2720f640 10 monclient(hunting): authenticate will time out at 1407.360359s 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c24fab640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f4c201422b0 0x7f4c2013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:39796/0 (socket says 172.21.15.19:39796) 2024-03-29T11:28:07.087 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.096+0000 7f4c24fab640 1 -- 172.21.15.19:0/763026559 learned_addr learned my addr 172.21.15.19:0/763026559 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845964129 0 0) 0x7f4c2013fbd0 con 0x7f4c20142690 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c00003520 con 0x7f4c20142690 2024-03-29T11:28:07.091 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655108166 0 0) 0x7f4c20142c80 con 0x7f4c201422b0 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c2013fbd0 con 0x7f4c201422b0 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1336637418 0 0) 0x7f4c2013fbd0 con 0x7f4c201422b0 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 >> v1:172.21.15.19:6790/0 conn(0x7f4c20142690 legacy=0x7f4c200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 >> v1:172.21.15.62:6789/0 conn(0x7f4c200b17d0 legacy=0x7f4c200b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:28:07.092 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c20151940 con 0x7f4c201422b0 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4c18001630 con 0x7f4c201422b0 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c1f7fe640 10 monclient: dump: 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:07.093 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c2720f640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.097+0000 7f4c2720f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c18001810 con 0x7f4c201422b0 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c1f7fe640 1 -- 172.21.15.19:0/763026559 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4c18001630 con 0x7f4c201422b0 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c1f7fe640 10 monclient: dump: 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:07.094 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c2720f640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.098+0000 7f4c2720f640 4 set_mon_vals no callback set 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 10 monclient: shutdown 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 1 -- 172.21.15.19:0/763026559 >> v1:172.21.15.19:6789/0 conn(0x7f4c201422b0 legacy=0x7f4c2013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:07.095 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 1 -- 172.21.15.19:0/763026559 shutdown_connections 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 1 -- 172.21.15.19:0/763026559 >> 172.21.15.19:0/763026559 conn(0x7f4c20066a00 msgr2=0x7f4c20066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 1 -- 172.21.15.19:0/763026559 shutdown_connections 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.099+0000 7f4c2720f640 1 -- 172.21.15.19:0/763026559 wait complete. 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.100+0000 7f4c2720f640 10 monclient: build_initial_monmap 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.100+0000 7f4c2720f640 10 monclient: monmap: 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:07.096 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:07.097 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:07.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.101+0000 7f4c2720f640 1 librados: starting msgr at 2024-03-29T11:28:07.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.101+0000 7f4c2720f640 1 librados: starting objecter 2024-03-29T11:28:07.097 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.101+0000 7f4c2720f640 1 Processor -- start 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.102+0000 7f4c2720f640 1 -- start start 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.102+0000 7f4c2720f640 1 librados: setting wanted keys 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.102+0000 7f4c2720f640 1 librados: calling monclient init 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.102+0000 7f4c2720f640 10 monclient: init 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.103+0000 7f4c2720f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.103+0000 7f4c2720f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.103+0000 7f4c2720f640 10 monclient(hunting): picked mon.noname-c con 0x7f4c200b17d0 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.103+0000 7f4c2720f640 10 monclient(hunting): picked mon.noname-a con 0x7f4c201422b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c2720f640 10 monclient(hunting): picked mon.noname-b con 0x7f4c20142690 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:07.105 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c2720f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200b56e0 con 0x7f4c201422b0 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c257ac640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f4c20142690 0x7f4c2012bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:37752/0 (socket says 172.21.15.19:37752) 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c257ac640 1 -- 172.21.15.19:0/2579035967 learned_addr learned my addr 172.21.15.19:0/2579035967 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 client.?.objecter ms_handle_connect 0x7f4c20142690 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 client.?.objecter ms_handle_connect 0x7f4c200b17d0 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200b6820 con 0x7f4c20142690 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c200b6a90 con 0x7f4c200b17d0 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c2720f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c2720f640 10 monclient(hunting): authenticate will time out at 1407.368652s 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 client.?.objecter ms_handle_connect 0x7f4c201422b0 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:07.106 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256567965 0 0) 0x7f4c200b6a90 con 0x7f4c200b17d0 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.104+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c04004eb0 con 0x7f4c200b17d0 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138905937 0 0) 0x7f4c200b56e0 con 0x7f4c201422b0 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c200b6a90 con 0x7f4c201422b0 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741509377 0 0) 0x7f4c200b6820 con 0x7f4c20142690 2024-03-29T11:28:07.107 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c200b56e0 con 0x7f4c20142690 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 948287482 0 0) 0x7f4c04004eb0 con 0x7f4c200b17d0 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c200b6820 con 0x7f4c200b17d0 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4c18001630 con 0x7f4c200b17d0 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.108 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.105+0000 7f4c1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1096611120 0 0) 0x7f4c200b6820 con 0x7f4c200b17d0 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 >> v1:172.21.15.62:6789/0 conn(0x7f4c20142690 legacy=0x7f4c2012bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 >> v1:172.21.15.19:6789/0 conn(0x7f4c201422b0 legacy=0x7f4c2012b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 monclient: found mon.c 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:07.109 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2012c650 con 0x7f4c200b17d0 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 10 monclient: _renew_subs 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c200b6d70 con 0x7f4c200b17d0 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 10 monclient: _renew_subs 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c200b5270 con 0x7f4c200b17d0 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c2720f640 1 librados: init done 2024-03-29T11:28:07.110 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c18004a90 con 0x7f4c200b17d0 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f4c18004da0 con 0x7f4c200b17d0 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1d7fa640 10 monclient: dump: 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:07.111 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.106+0000 7f4c1e7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.107+0000 7f4c2720f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.107+0000 7f4c2720f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.107+0000 7f4c2720f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.107+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bf8005080 con 0x7f4c200b17d0 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.108+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f4c1801ad70 con 0x7f4c200b17d0 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.109+0000 7f4c1ffff640 10 monclient: get_auth_request con 0x7f4c0406dbe0 auth_method 0 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.110+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f4c18084ba0 con 0x7f4c200b17d0 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.110+0000 7f4c1d7fa640 10 client.4571.objecter ms_dispatch 0x7f4c20000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.110+0000 7f4c1d7fa640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.110+0000 7f4c1d7fa640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.110+0000 7f4c1d7fa640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:28:07.112 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.110+0000 7f4c1d7fa640 10 client.4571.objecter ms_handle_connect 0x7f4c0406dbe0 2024-03-29T11:28:07.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.118+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c180504e0 con 0x7f4c200b17d0 2024-03-29T11:28:07.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.118+0000 7f4c1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:07.113 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.118+0000 7f4c1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:07.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.516+0000 7f4c2720f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:07.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.516+0000 7f4c2720f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:07.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.516+0000 7f4c2720f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:07.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.516+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bf8005330 con 0x7f4c200b17d0 2024-03-29T11:28:07.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.517+0000 7f4c1d7fa640 1 -- 172.21.15.19:0/2579035967 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f4c18084250 con 0x7f4c200b17d0 2024-03-29T11:28:07.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.518+0000 7f4c1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:07.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.518+0000 7f4c1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:07.508 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:07.508 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 10 librados: watch_flush enter 2024-03-29T11:28:07.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 10 librados: watch_flush exit 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 >> v1:172.21.15.19:6816/3721232097 conn(0x7f4c0406dbe0 legacy=0x7f4c040700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 10 monclient: shutdown 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.522+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 >> v1:172.21.15.19:6790/0 conn(0x7f4c200b17d0 legacy=0x7f4c200b7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:07.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.523+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 shutdown_connections 2024-03-29T11:28:07.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.523+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 >> 172.21.15.19:0/2579035967 conn(0x7f4c20066a00 msgr2=0x7f4c20132800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:07.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.523+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 shutdown_connections 2024-03-29T11:28:07.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.523+0000 7f4c2720f640 1 -- 172.21.15.19:0/2579035967 wait complete. 2024-03-29T11:28:07.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:07.523+0000 7f4c2720f640 1 librados: shutdown 2024-03-29T11:28:07.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:14.336 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:14.486 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.494+0000 7f914204e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:14.486 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.494+0000 7f914204e640 10 monclient: build_initial_monmap 2024-03-29T11:28:14.486 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.494+0000 7f914204e640 10 monclient: monmap: 2024-03-29T11:28:14.486 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:14.486 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:14.486 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:14.496190+0000 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:14.496190+0000 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:14.487 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:14.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.496+0000 7f914204e640 1 Processor -- start 2024-03-29T11:28:14.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.496+0000 7f914204e640 1 -- start start 2024-03-29T11:28:14.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.496+0000 7f914204e640 10 monclient: init 2024-03-29T11:28:14.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.497+0000 7f914204e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:14.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.497+0000 7f914204e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:14.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.497+0000 7f914204e640 10 monclient(hunting): picked mon.noname-b con 0x7f913c071de0 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.497+0000 7f914204e640 10 monclient(hunting): picked mon.noname-a con 0x7f913c0721c0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f914204e640 10 monclient(hunting): picked mon.noname-c con 0x7f913c145ae0 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f914204e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c145650 con 0x7f913c0721c0 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f914204e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0ae260 con 0x7f913c145ae0 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f914204e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c1420d0 con 0x7f913c071de0 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f914204e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f914204e640 10 monclient(hunting): authenticate will time out at 1414.762028s 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f913affd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f913c0721c0 0x7f913c13f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:37790/0 (socket says 172.21.15.19:37790) 2024-03-29T11:28:14.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.498+0000 7f913affd640 1 -- 172.21.15.19:0/1269461409 learned_addr learned my addr 172.21.15.19:0/1269461409 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:14.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688860150 0 0) 0x7f913c145650 con 0x7f913c0721c0 2024-03-29T11:28:14.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:14.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:14.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:14.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:28:14.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9124003520 con 0x7f913c0721c0 2024-03-29T11:28:14.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008341159 0 0) 0x7f9124003520 con 0x7f913c0721c0 2024-03-29T11:28:14.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.499+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 >> v1:172.21.15.19:6790/0 conn(0x7f913c071de0 legacy=0x7f913c0aed80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 >> v1:172.21.15.62:6789/0 conn(0x7f913c145ae0 legacy=0x7f913c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0b0590 con 0x7f913c0721c0 2024-03-29T11:28:14.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f912c001690 con 0x7f913c0721c0 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: dump: 2024-03-29T11:28:14.494 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:14.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f912c0017f0 con 0x7f913c0721c0 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.502+0000 7f913a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f913a7fc640 1 -- 172.21.15.19:0/1269461409 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f912c001c40 con 0x7f913c0721c0 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f913a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f913a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f913a7fc640 10 monclient: dump: 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:14.496 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:14.497 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:14.497 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:14.497 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:14.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f914204e640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:28:14.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f914204e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.503+0000 7f914204e640 4 set_mon_vals no callback set 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.504+0000 7f914204e640 10 monclient: shutdown 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.504+0000 7f914204e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.504+0000 7f914204e640 1 -- 172.21.15.19:0/1269461409 >> v1:172.21.15.19:6789/0 conn(0x7f913c0721c0 legacy=0x7f913c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.504+0000 7f914204e640 1 -- 172.21.15.19:0/1269461409 shutdown_connections 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.504+0000 7f914204e640 1 -- 172.21.15.19:0/1269461409 >> 172.21.15.19:0/1269461409 conn(0x7f913c06b660 msgr2=0x7f913c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.506+0000 7f914204e640 1 -- 172.21.15.19:0/1269461409 shutdown_connections 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.506+0000 7f914204e640 1 -- 172.21.15.19:0/1269461409 wait complete. 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.507+0000 7f914204e640 10 monclient: build_initial_monmap 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.507+0000 7f914204e640 10 monclient: monmap: 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:14.499 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:14.500 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 1 librados: starting msgr at 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 1 librados: starting objecter 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 1 Processor -- start 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 1 -- start start 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 1 librados: setting wanted keys 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 1 librados: calling monclient init 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.508+0000 7f914204e640 10 monclient: init 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:14.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient(hunting): picked mon.noname-c con 0x7f913c071de0 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient(hunting): picked mon.noname-a con 0x7f913c145ae0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient(hunting): picked mon.noname-b con 0x7f913c06d9d0 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c139fa0 con 0x7f913c145ae0 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c1db7d0 con 0x7f913c06d9d0 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c1dc400 con 0x7f913c071de0 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f914204e640 10 monclient(hunting): authenticate will time out at 1414.773409s 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f913affd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f913c145ae0 0x7f913c136e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:39830/0 (socket says 172.21.15.19:39830) 2024-03-29T11:28:14.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.509+0000 7f913affd640 1 -- 172.21.15.19:0/3495085516 learned_addr learned my addr 172.21.15.19:0/3495085516 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:14.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 10 client.?.objecter ms_handle_connect 0x7f913c145ae0 2024-03-29T11:28:14.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943457949 0 0) 0x7f913c139fa0 con 0x7f913c145ae0 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.512+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9128004eb0 con 0x7f913c145ae0 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 client.?.objecter ms_handle_connect 0x7f913c071de0 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178871186 0 0) 0x7f913c1dc400 con 0x7f913c071de0 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:14.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f913c139fa0 con 0x7f913c071de0 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469962292 0 0) 0x7f9128004eb0 con 0x7f913c145ae0 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f913c1dc400 con 0x7f913c145ae0 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f912c001c00 con 0x7f913c145ae0 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:14.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): dump: 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:14.507 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:14.508 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:14.508 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:14.508 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:14.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:14.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3792683336 0 0) 0x7f913c1dc400 con 0x7f913c145ae0 2024-03-29T11:28:14.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 >> v1:172.21.15.19:6790/0 conn(0x7f913c071de0 legacy=0x7f913c1dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:14.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.513+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 >> v1:172.21.15.62:6789/0 conn(0x7f913c06d9d0 legacy=0x7f913c070870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:14.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:14.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 1 monclient: found mon.a 2024-03-29T11:28:14.510 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c1dc590 con 0x7f913c145ae0 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.515+0000 7f9123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 5 monclient: authenticate success, global_id 4539 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 10 monclient: _renew_subs 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c137560 con 0x7f913c145ae0 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 10 client.4539.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 10 monclient: _renew_subs 2024-03-29T11:28:14.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c070f80 con 0x7f913c145ae0 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.517+0000 7f914204e640 1 librados: init done 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f912c004a90 con 0x7f913c145ae0 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9123fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f912c004e20 con 0x7f913c145ae0 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9123fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9123fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9123fff640 10 monclient: dump: 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:14.512 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.518+0000 7f9139ffb640 4 set_mon_vals no callback set 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.519+0000 7f9121ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.519+0000 7f9121ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.519+0000 7f9121ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:14.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.519+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c005080 con 0x7f913c145ae0 2024-03-29T11:28:14.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.522+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f912c004f80 con 0x7f913c145ae0 2024-03-29T11:28:14.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.523+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f912c084c40 con 0x7f913c145ae0 2024-03-29T11:28:14.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.523+0000 7f9123fff640 10 client.4539.objecter ms_dispatch 0x7f913c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:14.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.523+0000 7f9123fff640 3 client.4539.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:14.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.523+0000 7f9123fff640 3 client.4539.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:14.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.523+0000 7f9123fff640 20 client.4539.objecter dump_active .. 0 homeless 2024-03-29T11:28:14.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.524+0000 7f913b7fe640 10 monclient: get_auth_request con 0x7f912806dd20 auth_method 0 2024-03-29T11:28:14.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.524+0000 7f9123fff640 10 client.4539.objecter ms_handle_connect 0x7f912806dd20 2024-03-29T11:28:14.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.529+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f912c050520 con 0x7f913c145ae0 2024-03-29T11:28:14.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.529+0000 7f9123fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:14.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.529+0000 7f9123fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:14.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.868+0000 7f9121ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:14.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.868+0000 7f9121ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:14.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.868+0000 7f9121ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:14.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.868+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f911c005330 con 0x7f913c145ae0 2024-03-29T11:28:14.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.869+0000 7f9123fff640 1 -- 172.21.15.19:0/3495085516 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f912c052450 con 0x7f913c145ae0 2024-03-29T11:28:14.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.869+0000 7f9123fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:14.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.869+0000 7f9123fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:14.859 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:14.860 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:14.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.872+0000 7f9121ffb640 10 librados: watch_flush enter 2024-03-29T11:28:14.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.872+0000 7f9121ffb640 10 librados: watch_flush exit 2024-03-29T11:28:14.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.872+0000 7f9121ffb640 20 client.4539.objecter shutdown clearing up homeless session... 2024-03-29T11:28:14.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.872+0000 7f9121ffb640 10 client.4539.objecter successfully canceled tick 2024-03-29T11:28:14.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.873+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 >> v1:172.21.15.19:6816/3721232097 conn(0x7f912806dd20 legacy=0x7f91280701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:14.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.873+0000 7f9121ffb640 10 monclient: shutdown 2024-03-29T11:28:14.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.873+0000 7f9121ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:14.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.873+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 >> v1:172.21.15.19:6789/0 conn(0x7f913c145ae0 legacy=0x7f913c136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:14.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.873+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 shutdown_connections 2024-03-29T11:28:14.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.873+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 >> 172.21.15.19:0/3495085516 conn(0x7f913c06b660 msgr2=0x7f913c0a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:14.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.874+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 shutdown_connections 2024-03-29T11:28:14.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.874+0000 7f9121ffb640 1 -- 172.21.15.19:0/3495085516 wait complete. 2024-03-29T11:28:14.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:14.874+0000 7f9121ffb640 1 librados: shutdown 2024-03-29T11:28:14.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:21.693 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:21.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.840+0000 7fafdd90e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:21.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.840+0000 7fafdd90e640 10 monclient: build_initial_monmap 2024-03-29T11:28:21.831 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.840+0000 7fafdd90e640 10 monclient: monmap: 2024-03-29T11:28:21.831 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:21.831 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:21.842240+0000 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:21.842240+0000 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.841+0000 7fafdd90e640 1 Processor -- start 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.841+0000 7fafdd90e640 1 -- start start 2024-03-29T11:28:21.832 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.841+0000 7fafdd90e640 10 monclient: init 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.841+0000 7fafdd90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.841+0000 7fafdd90e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.841+0000 7fafdd90e640 10 monclient(hunting): picked mon.noname-b con 0x7fafd80a9750 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 10 monclient(hunting): picked mon.noname-a con 0x7fafd813e1e0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 10 monclient(hunting): picked mon.noname-c con 0x7fafd8071c30 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80ab300 con 0x7fafd813e1e0 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80ac2f0 con 0x7fafd8071c30 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd6ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fafd80a9750 0x7fafd813dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:39342/0 (socket says 172.21.15.19:39342) 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd6ffd640 1 -- 172.21.15.19:0/3078140984 learned_addr learned my addr 172.21.15.19:0/3078140984 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 1 -- 172.21.15.19:0/3078140984 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80ad8d0 con 0x7fafd80a9750 2024-03-29T11:28:21.834 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafdd90e640 10 monclient(hunting): authenticate will time out at 1422.106319s 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266855404 0 0) 0x7fafd80ad8d0 con 0x7fafd80a9750 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc0003440 con 0x7fafd80a9750 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 461755631 0 0) 0x7fafc0003440 con 0x7fafd80a9750 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 >> v1:172.21.15.62:6789/0 conn(0x7fafd8071c30 legacy=0x7fafd80ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.842+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 >> v1:172.21.15.19:6789/0 conn(0x7fafd813e1e0 legacy=0x7fafd80ac710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.844+0000 7fafd5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.844+0000 7fafd5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.844+0000 7fafd5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80ada60 con 0x7fafd80a9750 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fafc8001690 con 0x7fafd80a9750 2024-03-29T11:28:21.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: dump: 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.838 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafdd90e640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafdd90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafc8001920 con 0x7fafd80a9750 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 1 -- 172.21.15.19:0/3078140984 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fafc8001cb0 con 0x7fafd80a9750 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafd5ffb640 10 monclient: dump: 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:21.839 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafdd90e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.845+0000 7fafdd90e640 4 set_mon_vals no callback set 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 10 monclient: shutdown 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 1 -- 172.21.15.19:0/3078140984 >> v1:172.21.15.19:6790/0 conn(0x7fafd80a9750 legacy=0x7fafd813dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 1 -- 172.21.15.19:0/3078140984 shutdown_connections 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 1 -- 172.21.15.19:0/3078140984 >> 172.21.15.19:0/3078140984 conn(0x7fafd806b660 msgr2=0x7fafd806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 1 -- 172.21.15.19:0/3078140984 shutdown_connections 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.846+0000 7fafdd90e640 1 -- 172.21.15.19:0/3078140984 wait complete. 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.847+0000 7fafdd90e640 10 monclient: build_initial_monmap 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.847+0000 7fafdd90e640 10 monclient: monmap: 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:21.841 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.847+0000 7fafdd90e640 1 librados: starting msgr at 2024-03-29T11:28:21.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.847+0000 7fafdd90e640 1 librados: starting objecter 2024-03-29T11:28:21.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.847+0000 7fafdd90e640 1 Processor -- start 2024-03-29T11:28:21.847 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.848+0000 7fafdd90e640 1 -- start start 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.848+0000 7fafdd90e640 1 librados: setting wanted keys 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.848+0000 7fafdd90e640 1 librados: calling monclient init 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.848+0000 7fafdd90e640 10 monclient: init 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.848+0000 7fafdd90e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.849+0000 7fafdd90e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.849+0000 7fafdd90e640 10 monclient(hunting): picked mon.noname-b con 0x7fafd8071c30 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 10 monclient(hunting): picked mon.noname-a con 0x7fafd80a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafd6ffd640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fafd8071c30 0x7fafd806dc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:55462/0 (socket says 172.21.15.19:55462) 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafd6ffd640 1 -- 172.21.15.19:0/2373456559 learned_addr learned my addr 172.21.15.19:0/2373456559 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 10 monclient(hunting): picked mon.noname-c con 0x7fafd813e1e0 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd81429c0 con 0x7fafd80a9750 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 10 client.?.objecter ms_handle_connect 0x7fafd8071c30 2024-03-29T11:28:21.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd81426e0 con 0x7fafd8071c30 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd806f600 con 0x7fafd813e1e0 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafdd90e640 10 monclient(hunting): authenticate will time out at 1422.114606s 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374103296 0 0) 0x7fafd81426e0 con 0x7fafd8071c30 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:21.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.850+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4004eb0 con 0x7fafd8071c30 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2372371403 0 0) 0x7fafc4004eb0 con 0x7fafd8071c30 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafd81426e0 con 0x7fafd8071c30 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 client.?.objecter ms_handle_connect 0x7fafd80a9750 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fafc8001610 con 0x7fafd8071c30 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 monclient(hunting): dump: 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.850 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 client.?.objecter ms_handle_connect 0x7fafd813e1e0 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.851+0000 7fafbffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 507509469 0 0) 0x7fafd81429c0 con 0x7fafd80a9750 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:21.851 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc4004eb0 con 0x7fafd80a9750 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296342296 0 0) 0x7fafd806f600 con 0x7fafd813e1e0 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd81429c0 con 0x7fafd813e1e0 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3206205802 0 0) 0x7fafc4004eb0 con 0x7fafd80a9750 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafc4005b70 con 0x7fafd80a9750 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fafcc0016f0 con 0x7fafd80a9750 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:21.852 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): dump: 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001385214 0 0) 0x7fafd81429c0 con 0x7fafd813e1e0 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafc4004eb0 con 0x7fafd813e1e0 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fafd0001ac0 con 0x7fafd813e1e0 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:21.853 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 10 monclient(hunting): dump: 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1190328520 0 0) 0x7fafc4004eb0 con 0x7fafd813e1e0 2024-03-29T11:28:21.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.852+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 >> v1:172.21.15.62:6789/0 conn(0x7fafd8071c30 legacy=0x7fafd806dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 >> v1:172.21.15.19:6789/0 conn(0x7fafd80a9750 legacy=0x7fafd8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 1 monclient: found mon.c 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd806f870 con 0x7fafd813e1e0 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd0001f90 con 0x7fafd813e1e0 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafd57fa640 4 set_mon_vals no callback set 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fafd0001ac0 con 0x7fafd813e1e0 2024-03-29T11:28:21.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafbffff640 10 monclient: dump: 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:28:21.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 10 monclient: _renew_subs 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd806e350 con 0x7fafd813e1e0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 10 client.4580.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 10 monclient: _renew_subs 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd81e7550 con 0x7fafd813e1e0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.853+0000 7fafdd90e640 1 librados: init done 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.855+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fafd00197e0 con 0x7fafd813e1e0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.855+0000 7fafdd90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.855+0000 7fafdd90e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.855+0000 7fafdd90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.855+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd8075f10 con 0x7fafd813e1e0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.856+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fafd0083340 con 0x7fafd813e1e0 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.856+0000 7fafbffff640 10 client.4580.objecter ms_dispatch 0x7fafd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:21.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.856+0000 7fafbffff640 3 client.4580.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:21.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.856+0000 7fafbffff640 3 client.4580.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:21.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.856+0000 7fafbffff640 20 client.4580.objecter dump_active .. 0 homeless 2024-03-29T11:28:21.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.857+0000 7fafd6ffd640 10 monclient: get_auth_request con 0x7fafc406e660 auth_method 0 2024-03-29T11:28:21.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.857+0000 7fafbffff640 10 client.4580.objecter ms_handle_connect 0x7fafc406e660 2024-03-29T11:28:21.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.862+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafd004ed60 con 0x7fafd813e1e0 2024-03-29T11:28:21.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.862+0000 7fafbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:21.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:21.862+0000 7fafbffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:22.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.175+0000 7fafdd90e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:22.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.175+0000 7fafdd90e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:22.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.175+0000 7fafdd90e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:22.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.175+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafd806ff70 con 0x7fafd813e1e0 2024-03-29T11:28:22.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.176+0000 7fafbffff640 1 -- 172.21.15.19:0/2373456559 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fafd0050c90 con 0x7fafd813e1e0 2024-03-29T11:28:22.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.176+0000 7fafbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:22.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.176+0000 7fafbffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:22.166 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:22.166 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 10 librados: watch_flush enter 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 10 librados: watch_flush exit 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 20 client.4580.objecter shutdown clearing up homeless session... 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 10 client.4580.objecter successfully canceled tick 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 >> v1:172.21.15.19:6816/3721232097 conn(0x7fafc406e660 legacy=0x7fafc4070b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 10 monclient: shutdown 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.179+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 >> v1:172.21.15.19:6790/0 conn(0x7fafd813e1e0 legacy=0x7fafd8070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.180+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 shutdown_connections 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.180+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 >> 172.21.15.19:0/2373456559 conn(0x7fafd806b660 msgr2=0x7fafd80ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:22.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.180+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 shutdown_connections 2024-03-29T11:28:22.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.180+0000 7fafdd90e640 1 -- 172.21.15.19:0/2373456559 wait complete. 2024-03-29T11:28:22.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:22.180+0000 7fafdd90e640 1 librados: shutdown 2024-03-29T11:28:22.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:28.989 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:29.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.152+0000 7ff202f15640 10 monclient: get_monmap_and_config 2024-03-29T11:28:29.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.152+0000 7ff202f15640 10 monclient: build_initial_monmap 2024-03-29T11:28:29.143 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.152+0000 7ff202f15640 10 monclient: monmap: 2024-03-29T11:28:29.143 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:29.153257+0000 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:29.153257+0000 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:29.144 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.153+0000 7ff202f15640 1 Processor -- start 2024-03-29T11:28:29.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.154+0000 7ff202f15640 1 -- start start 2024-03-29T11:28:29.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.154+0000 7ff202f15640 10 monclient: init 2024-03-29T11:28:29.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.155+0000 7ff202f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:29.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.155+0000 7ff202f15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:29.145 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.155+0000 7ff202f15640 10 monclient(hunting): picked mon.noname-b con 0x7ff1fc0b4100 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:29.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.155+0000 7ff202f15640 10 monclient(hunting): picked mon.noname-a con 0x7ff1fc13ee90 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:29.146 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.155+0000 7ff200cb1640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff1fc0b4100 0x7ff1fc0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:56670/0 (socket says 172.21.15.19:56670) 2024-03-29T11:28:29.147 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.157+0000 7ff200cb1640 1 -- 172.21.15.19:0/2954054108 learned_addr learned my addr 172.21.15.19:0/2954054108 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:29.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.158+0000 7ff202f15640 10 monclient(hunting): picked mon.noname-c con 0x7ff1fc0af900 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:29.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.158+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0afd10 con 0x7ff1fc13ee90 2024-03-29T11:28:29.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.158+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0af6f0 con 0x7ff1fc0af900 2024-03-29T11:28:29.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.158+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0aac80 con 0x7ff1fc0b4100 2024-03-29T11:28:29.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.158+0000 7ff202f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:29.150 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.158+0000 7ff202f15640 10 monclient(hunting): authenticate will time out at 1429.421594s 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613620785 0 0) 0x7ff1fc0aac80 con 0x7ff1fc0b4100 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4005730 con 0x7ff1fc0b4100 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3749211311 0 0) 0x7ff1e4005730 con 0x7ff1fc0b4100 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 >> v1:172.21.15.62:6789/0 conn(0x7ff1fc0af900 legacy=0x7ff1fc14ac10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 >> v1:172.21.15.19:6789/0 conn(0x7ff1fc13ee90 legacy=0x7ff1fc13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0aaf60 con 0x7ff1fc0b4100 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff1f4001a10 con 0x7ff1fc0b4100 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff1fb7fe640 10 monclient: dump: 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:29.152 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff202f15640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.160+0000 7ff202f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f4001ca0 con 0x7ff1fc0b4100 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff1fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff1fb7fe640 1 -- 172.21.15.19:0/2954054108 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff1f4002030 con 0x7ff1fc0b4100 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff1fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:29.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff1fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff1fb7fe640 10 monclient: dump: 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff202f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:29.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff202f15640 4 set_mon_vals no callback set 2024-03-29T11:28:29.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff202f15640 10 monclient: shutdown 2024-03-29T11:28:29.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff202f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:29.155 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.161+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 >> v1:172.21.15.19:6790/0 conn(0x7ff1fc0b4100 legacy=0x7ff1fc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.162+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 shutdown_connections 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.162+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 >> 172.21.15.19:0/2954054108 conn(0x7ff1fc0b35e0 msgr2=0x7ff1fc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.162+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 shutdown_connections 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.163+0000 7ff202f15640 1 -- 172.21.15.19:0/2954054108 wait complete. 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.164+0000 7ff202f15640 10 monclient: build_initial_monmap 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.164+0000 7ff202f15640 10 monclient: monmap: 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:29.156 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:29.157 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:29.157 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:29.157 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:29.158 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:29.158 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:29.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 1 librados: starting msgr at 2024-03-29T11:28:29.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 1 librados: starting objecter 2024-03-29T11:28:29.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 1 Processor -- start 2024-03-29T11:28:29.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 1 -- start start 2024-03-29T11:28:29.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 1 librados: setting wanted keys 2024-03-29T11:28:29.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 1 librados: calling monclient init 2024-03-29T11:28:29.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.165+0000 7ff202f15640 10 monclient: init 2024-03-29T11:28:29.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:29.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:29.159 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient(hunting): picked mon.noname-a con 0x7ff1fc0af900 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient(hunting): picked mon.noname-b con 0x7ff1fc0b4100 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient(hunting): picked mon.noname-c con 0x7ff1fc13ee90 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc13a260 con 0x7ff1fc0af900 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc13a4b0 con 0x7ff1fc0b4100 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc137f80 con 0x7ff1fc13ee90 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.166+0000 7ff202f15640 10 monclient(hunting): authenticate will time out at 1429.429543s 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.167+0000 7ff2014b2640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff1fc13ee90 0x7ff1fc1de870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:37806/0 (socket says 172.21.15.19:37806) 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.167+0000 7ff2014b2640 1 -- 172.21.15.19:0/3844179373 learned_addr learned my addr 172.21.15.19:0/3844179373 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.169+0000 7ff1f97fa640 10 client.?.objecter ms_handle_connect 0x7ff1fc13ee90 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.169+0000 7ff1f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816910895 0 0) 0x7ff1fc137f80 con 0x7ff1fc13ee90 2024-03-29T11:28:29.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:29.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:29.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:29.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:28:29.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e8004eb0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1484725238 0 0) 0x7ff1e8004eb0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1fc137f80 con 0x7ff1fc13ee90 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff1ec0017f0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): dump: 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:29.162 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.170+0000 7ff1f97fa640 10 client.?.objecter ms_handle_connect 0x7ff1fc0af900 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 client.?.objecter ms_handle_connect 0x7ff1fc0b4100 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:29.163 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527056244 0 0) 0x7ff1fc13a260 con 0x7ff1fc0af900 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e8004eb0 con 0x7ff1fc0af900 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 88548626 0 0) 0x7ff1fc137f80 con 0x7ff1fc13ee90 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 >> v1:172.21.15.62:6789/0 conn(0x7ff1fc0b4100 legacy=0x7ff1fc1d64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 >> v1:172.21.15.19:6789/0 conn(0x7ff1fc0af900 legacy=0x7ff1fc136850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 monclient: found mon.c 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:29.164 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc137410 con 0x7ff1fc13ee90 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff1f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 10 monclient: _renew_subs 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc136f60 con 0x7ff1fc13ee90 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 10 monclient: _renew_subs 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:29.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc13aab0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.171+0000 7ff202f15640 1 librados: init done 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.173+0000 7ff202f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.173+0000 7ff202f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.173+0000 7ff202f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.173+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc137de0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.174+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1ec001950 con 0x7ff1fc13ee90 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.174+0000 7ff1f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.174+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff1ec004900 con 0x7ff1fc13ee90 2024-03-29T11:28:29.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.174+0000 7ff1f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.174+0000 7ff1f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.174+0000 7ff1f97fa640 10 monclient: dump: 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.175+0000 7ff1faffd640 4 set_mon_vals no callback set 2024-03-29T11:28:29.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.176+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff1ec01a8d0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.176+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff1ec0847f0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.177+0000 7ff1f97fa640 10 client.4586.objecter ms_dispatch 0x7ff1fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.177+0000 7ff1f97fa640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.177+0000 7ff1f97fa640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.177+0000 7ff200cb1640 10 monclient: get_auth_request con 0x7ff1e806de20 auth_method 0 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.177+0000 7ff1f97fa640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:28:29.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.177+0000 7ff1f97fa640 10 client.4586.objecter ms_handle_connect 0x7ff1e806de20 2024-03-29T11:28:29.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.182+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1ec0500d0 con 0x7ff1fc13ee90 2024-03-29T11:28:29.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.182+0000 7ff1f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:29.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.182+0000 7ff1f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:29.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.510+0000 7ff202f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:29.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.510+0000 7ff202f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:29.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.510+0000 7ff202f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:29.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.510+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1fc138110 con 0x7ff1fc13ee90 2024-03-29T11:28:29.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.511+0000 7ff1f97fa640 1 -- 172.21.15.19:0/3844179373 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff1ec052000 con 0x7ff1fc13ee90 2024-03-29T11:28:29.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.511+0000 7ff1f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:29.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.511+0000 7ff1f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:29.502 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:29.502 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:29.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 10 librados: watch_flush enter 2024-03-29T11:28:29.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 10 librados: watch_flush exit 2024-03-29T11:28:29.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:28:29.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:28:29.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff1e806de20 legacy=0x7ff1e80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 10 monclient: shutdown 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.516+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 >> v1:172.21.15.19:6790/0 conn(0x7ff1fc13ee90 legacy=0x7ff1fc1de870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.517+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 shutdown_connections 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.517+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 >> 172.21.15.19:0/3844179373 conn(0x7ff1fc0b35e0 msgr2=0x7ff1fc0acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.517+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 shutdown_connections 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.517+0000 7ff202f15640 1 -- 172.21.15.19:0/3844179373 wait complete. 2024-03-29T11:28:29.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:29.517+0000 7ff202f15640 1 librados: shutdown 2024-03-29T11:28:29.523 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:36.330 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:36.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.491+0000 7fb802a0f640 10 monclient: get_monmap_and_config 2024-03-29T11:28:36.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.491+0000 7fb802a0f640 10 monclient: build_initial_monmap 2024-03-29T11:28:36.481 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.491+0000 7fb802a0f640 10 monclient: monmap: 2024-03-29T11:28:36.481 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:36.481 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:36.481 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:36.492410+0000 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:36.492410+0000 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:36.482 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.485 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.495+0000 7fb802a0f640 1 Processor -- start 2024-03-29T11:28:36.488 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 1 -- start start 2024-03-29T11:28:36.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 10 monclient: init 2024-03-29T11:28:36.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:36.489 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:36.490 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7fc0b4890 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc13c040 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.499+0000 7fb802a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7fc1427e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.500+0000 7fb802a0f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0b42f0 con 0x7fb7fc0b4890 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.500+0000 7fb802a0f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0aa7a0 con 0x7fb7fc1427e0 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.500+0000 7fb802a0f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0aaa10 con 0x7fb7fc13c040 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.500+0000 7fb802a0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:36.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.500+0000 7fb802a0f640 10 monclient(hunting): authenticate will time out at 1436.763021s 2024-03-29T11:28:36.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.503+0000 7fb7fb7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fb7fc13c040 0x7fb7fc1420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60199/0 (socket says 172.21.15.19:60199) 2024-03-29T11:28:36.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.503+0000 7fb7fb7fe640 1 -- 172.21.15.19:0/1677653172 learned_addr learned my addr 172.21.15.19:0/1677653172 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:36.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.503+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334245988 0 0) 0x7fb7fc0aaa10 con 0x7fb7fc13c040 2024-03-29T11:28:36.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.504+0000 7fb7faffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:36.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.504+0000 7fb7faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:36.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.504+0000 7fb7faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:36.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.504+0000 7fb7faffd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:28:36.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.504+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e4003520 con 0x7fb7fc13c040 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1908000104 0 0) 0x7fb7e4003520 con 0x7fb7fc13c040 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 >> v1:172.21.15.62:6789/0 conn(0x7fb7fc1427e0 legacy=0x7fb7fc14d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 >> v1:172.21.15.19:6789/0 conn(0x7fb7fc0b4890 legacy=0x7fb7fc13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 monclient: found mon.noname-b 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0aacf0 con 0x7fb7fc13c040 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:36.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f0001690 con 0x7fb7fc13c040 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: dump: 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:36.497 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7f00018b0 con 0x7fb7fc13c040 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 1 -- 172.21.15.19:0/1677653172 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f0001c40 con 0x7fb7fc13c040 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.506+0000 7fb7faffd640 10 monclient: dump: 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.498 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:36.499 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.499 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:36.499 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:36.499 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:36.499 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.507+0000 7fb802a0f640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.507+0000 7fb802a0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.507+0000 7fb802a0f640 4 set_mon_vals no callback set 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.507+0000 7fb802a0f640 10 monclient: shutdown 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.508+0000 7fb802a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.508+0000 7fb802a0f640 1 -- 172.21.15.19:0/1677653172 >> v1:172.21.15.19:6790/0 conn(0x7fb7fc13c040 legacy=0x7fb7fc1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.508+0000 7fb802a0f640 1 -- 172.21.15.19:0/1677653172 shutdown_connections 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.508+0000 7fb802a0f640 1 -- 172.21.15.19:0/1677653172 >> 172.21.15.19:0/1677653172 conn(0x7fb7fc06d180 msgr2=0x7fb7fc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.508+0000 7fb802a0f640 1 -- 172.21.15.19:0/1677653172 shutdown_connections 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.508+0000 7fb802a0f640 1 -- 172.21.15.19:0/1677653172 wait complete. 2024-03-29T11:28:36.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.509+0000 7fb802a0f640 10 monclient: build_initial_monmap 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.509+0000 7fb802a0f640 10 monclient: monmap: 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.509+0000 7fb802a0f640 1 librados: starting msgr at 2024-03-29T11:28:36.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.509+0000 7fb802a0f640 1 librados: starting objecter 2024-03-29T11:28:36.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.509+0000 7fb802a0f640 1 Processor -- start 2024-03-29T11:28:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 1 -- start start 2024-03-29T11:28:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 1 librados: setting wanted keys 2024-03-29T11:28:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 1 librados: calling monclient init 2024-03-29T11:28:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient: init 2024-03-29T11:28:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:36.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient(hunting): picked mon.noname-a con 0x7fb7fc13c040 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc1427e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7fc0b0ba0 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc0b1640 con 0x7fb7fc13c040 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc1db950 con 0x7fb7fc1427e0 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7fc1db570 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:36.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.510+0000 7fb802a0f640 10 monclient(hunting): authenticate will time out at 1436.773808s 2024-03-29T11:28:36.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.515+0000 7fb7fbfff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fb7fc13c040 0x7fb7fc1deab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:60741/0 (socket says 172.21.15.19:60741) 2024-03-29T11:28:36.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.515+0000 7fb7fbfff640 1 -- 172.21.15.19:0/534553688 learned_addr learned my addr 172.21.15.19:0/534553688 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:36.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7fc13c040 2024-03-29T11:28:36.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:36.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7fc0b0ba0 2024-03-29T11:28:36.508 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642067204 0 0) 0x7fb7fc0b1640 con 0x7fb7fc13c040 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.517+0000 7fb7f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7e8004eb0 con 0x7fb7fc13c040 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001158972 0 0) 0x7fb7fc1db570 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:28:36.509 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.518+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc0b1640 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.513 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 client.?.objecter ms_handle_connect 0x7fb7fc1427e0 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577701538 0 0) 0x7fb7fc1db950 con 0x7fb7fc1427e0 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7fc1db570 con 0x7fb7fc1427e0 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4095694509 0 0) 0x7fb7fc0b1640 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7fc1db950 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3443729334 0 0) 0x7fb7e8004eb0 con 0x7fb7fc13c040 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7fc0b1640 con 0x7fb7fc13c040 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f4001bb0 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.514 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:36.515 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7ec001830 con 0x7fb7fc13c040 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.522+0000 7fb7f8ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:36.516 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3455429509 0 0) 0x7fb7fc1db950 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 >> v1:172.21.15.62:6789/0 conn(0x7fb7fc1427e0 legacy=0x7fb7fc1dbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 >> v1:172.21.15.19:6789/0 conn(0x7fb7fc13c040 legacy=0x7fb7fc1deab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 1 monclient: found mon.c 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc1db700 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb7f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.523+0000 7fb802a0f640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.524+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7f4003990 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.517 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.524+0000 7fb7f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:36.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.524+0000 7fb7fa7fc640 4 set_mon_vals no callback set 2024-03-29T11:28:36.518 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fb7f4003d20 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb7f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb7f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb7f8ff9640 10 monclient: dump: 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:36.519 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 10 monclient: _renew_subs 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7fc145ec0 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 10 monclient: _renew_subs 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7fc0b0370 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.520 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.525+0000 7fb802a0f640 1 librados: init done 2024-03-29T11:28:36.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.529+0000 7fb802a0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:36.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.529+0000 7fb802a0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:36.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.529+0000 7fb802a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:36.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.529+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7fc0ab7e0 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.521 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.530+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fb7f4003e80 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.531+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fb7f4083aa0 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.531+0000 7fb7f8ff9640 10 client.4592.objecter ms_dispatch 0x7fb7fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:36.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.531+0000 7fb7f8ff9640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:36.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.531+0000 7fb7f8ff9640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:36.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.531+0000 7fb7f8ff9640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:28:36.522 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.532+0000 7fb7fbfff640 10 monclient: get_auth_request con 0x7fb7e806dcb0 auth_method 0 2024-03-29T11:28:36.523 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.533+0000 7fb7f8ff9640 10 client.4592.objecter ms_handle_connect 0x7fb7e806dcb0 2024-03-29T11:28:36.530 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.540+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7f404f380 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.541+0000 7fb7f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:36.531 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.541+0000 7fb7f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:36.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.873+0000 7fb802a0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:36.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.873+0000 7fb802a0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:36.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.873+0000 7fb802a0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:36.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.873+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7fc1dbae0 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.875+0000 7fb7f8ff9640 1 -- 172.21.15.19:0/534553688 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fb7f40512b0 con 0x7fb7fc0b0ba0 2024-03-29T11:28:36.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.875+0000 7fb7f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:36.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.875+0000 7fb7f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:36.867 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:36.867 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 10 librados: watch_flush enter 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 10 librados: watch_flush exit 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 >> v1:172.21.15.19:6816/3721232097 conn(0x7fb7e806dcb0 legacy=0x7fb7e8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 10 monclient: shutdown 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 >> v1:172.21.15.19:6790/0 conn(0x7fb7fc0b0ba0 legacy=0x7fb7fc0afc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:36.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 shutdown_connections 2024-03-29T11:28:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.879+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 >> 172.21.15.19:0/534553688 conn(0x7fb7fc06d180 msgr2=0x7fb7fc0b8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.880+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 shutdown_connections 2024-03-29T11:28:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.880+0000 7fb802a0f640 1 -- 172.21.15.19:0/534553688 wait complete. 2024-03-29T11:28:36.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:36.880+0000 7fb802a0f640 1 librados: shutdown 2024-03-29T11:28:36.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:43.700 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:43.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.871+0000 7fec54b0e640 10 monclient: get_monmap_and_config 2024-03-29T11:28:43.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.871+0000 7fec54b0e640 10 monclient: build_initial_monmap 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.871+0000 7fec54b0e640 10 monclient: monmap: 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:43.872504+0000 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:43.872504+0000 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:43.862 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:43.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.872+0000 7fec54b0e640 1 Processor -- start 2024-03-29T11:28:43.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.872+0000 7fec54b0e640 1 -- start start 2024-03-29T11:28:43.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.872+0000 7fec54b0e640 10 monclient: init 2024-03-29T11:28:43.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:43.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:43.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fec501457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:43.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fec5013f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:43.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fec500a9d70 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:43.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec500aaef0 con 0x7fec500a9d70 2024-03-29T11:28:43.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec50145bd0 con 0x7fec501457f0 2024-03-29T11:28:43.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec50145eb0 con 0x7fec5013f650 2024-03-29T11:28:43.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:43.865 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.874+0000 7fec54b0e640 10 monclient(hunting): authenticate will time out at 1444.137795s 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4e59c640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fec501457f0 0x7fec5013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:58174/0 (socket says 172.21.15.19:58174) 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4e59c640 1 -- 172.21.15.19:0/2483440443 learned_addr learned my addr 172.21.15.19:0/2483440443 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660360765 0 0) 0x7fec50145bd0 con 0x7fec501457f0 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:28:43.867 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec38003520 con 0x7fec501457f0 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197269954 0 0) 0x7fec500aaef0 con 0x7fec500a9d70 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.875+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50145bd0 con 0x7fec500a9d70 2024-03-29T11:28:43.868 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808011816 0 0) 0x7fec50145eb0 con 0x7fec5013f650 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec380039d0 con 0x7fec5013f650 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 274732492 0 0) 0x7fec38003520 con 0x7fec501457f0 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 >> v1:172.21.15.19:6790/0 conn(0x7fec5013f650 legacy=0x7fec500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.869 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 >> v1:172.21.15.19:6789/0 conn(0x7fec500a9d70 legacy=0x7fec5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 monclient: found mon.noname-c 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50146340 con 0x7fec501457f0 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: _finish_auth 0 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fec44001690 con 0x7fec501457f0 2024-03-29T11:28:43.870 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: dump: 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec54b0e640 5 monclient: authenticate success, global_id 4537 2024-03-29T11:28:43.871 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec54b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:43.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec440017f0 con 0x7fec501457f0 2024-03-29T11:28:43.872 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:43.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec54b0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:43.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.876+0000 7fec54b0e640 4 set_mon_vals no callback set 2024-03-29T11:28:43.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.877+0000 7fec4d59a640 1 -- 172.21.15.19:0/2483440443 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fec44001c00 con 0x7fec501457f0 2024-03-29T11:28:43.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.877+0000 7fec4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.873 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.877+0000 7fec4d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.877+0000 7fec4d59a640 10 monclient: dump: 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:43.874 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:43.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 10 monclient: shutdown 2024-03-29T11:28:43.881 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 1 -- 172.21.15.19:0/2483440443 >> v1:172.21.15.62:6789/0 conn(0x7fec501457f0 legacy=0x7fec5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 1 -- 172.21.15.19:0/2483440443 shutdown_connections 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 1 -- 172.21.15.19:0/2483440443 >> 172.21.15.19:0/2483440443 conn(0x7fec50066a00 msgr2=0x7fec50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 1 -- 172.21.15.19:0/2483440443 shutdown_connections 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.879+0000 7fec54b0e640 1 -- 172.21.15.19:0/2483440443 wait complete. 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 10 monclient: build_initial_monmap 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 10 monclient: monmap: 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:43.882 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 1 librados: starting msgr at 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 1 librados: starting objecter 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 1 Processor -- start 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 1 -- start start 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 1 librados: setting wanted keys 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 1 librados: calling monclient init 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.881+0000 7fec54b0e640 10 monclient: init 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.882+0000 7fec54b0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:43.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.882+0000 7fec54b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fec500a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fec5013f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fec501457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5013bca0 con 0x7fec501457f0 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5013a130 con 0x7fec500a9d70 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5013b050 con 0x7fec5013f650 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec54b0e640 10 monclient(hunting): authenticate will time out at 1444.146118s 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec4dd9b640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fec5013f650 0x7fec50140740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:37722/0 (socket says 172.21.15.19:37722) 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec4dd9b640 1 -- 172.21.15.19:0/1661148420 learned_addr learned my addr 172.21.15.19:0/1661148420 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec3effd640 10 client.?.objecter ms_handle_connect 0x7fec500a9d70 2024-03-29T11:28:43.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.883+0000 7fec3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 client.?.objecter ms_handle_connect 0x7fec5013f650 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 client.?.objecter ms_handle_connect 0x7fec501457f0 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060945790 0 0) 0x7fec5013a130 con 0x7fec500a9d70 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.884+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec30004eb0 con 0x7fec500a9d70 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864805299 0 0) 0x7fec5013bca0 con 0x7fec501457f0 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5013a130 con 0x7fec501457f0 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1248958531 0 0) 0x7fec30004eb0 con 0x7fec500a9d70 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec5013bca0 con 0x7fec500a9d70 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294940489 0 0) 0x7fec5013b050 con 0x7fec5013f650 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec30004eb0 con 0x7fec5013f650 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fec44001b00 con 0x7fec500a9d70 2024-03-29T11:28:43.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): dump: 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:43.887 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3792005693 0 0) 0x7fec5013bca0 con 0x7fec500a9d70 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 >> v1:172.21.15.19:6790/0 conn(0x7fec5013f650 legacy=0x7fec50140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 >> v1:172.21.15.19:6789/0 conn(0x7fec501457f0 legacy=0x7fec50140e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.885+0000 7fec3effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 1 monclient: found mon.b 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5013b330 con 0x7fec500a9d70 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 10 monclient: _renew_subs 2024-03-29T11:28:43.888 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:28:43.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5013b750 con 0x7fec500a9d70 2024-03-29T11:28:43.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:43.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 10 monclient: _renew_subs 2024-03-29T11:28:43.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:28:43.889 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec44001c60 con 0x7fec500a9d70 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec501e3210 con 0x7fec500a9d70 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec54b0e640 1 librados: init done 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fec44004c30 con 0x7fec500a9d70 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec4cd99640 4 set_mon_vals no callback set 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.886+0000 7fec3effd640 10 monclient: dump: 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:43.890 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.888+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fec4401ac00 con 0x7fec500a9d70 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.888+0000 7fec54b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.888+0000 7fec54b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.888+0000 7fec54b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.888+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec50075f10 con 0x7fec500a9d70 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fec44084a10 con 0x7fec500a9d70 2024-03-29T11:28:43.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec3effd640 10 client.4540.objecter ms_dispatch 0x7fec50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:43.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec3effd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:43.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec3effd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:43.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec3effd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:28:43.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec4dd9b640 10 monclient: get_auth_request con 0x7fec3006dea0 auth_method 0 2024-03-29T11:28:43.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.890+0000 7fec3effd640 10 client.4540.objecter ms_handle_connect 0x7fec3006dea0 2024-03-29T11:28:43.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.895+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec440502f0 con 0x7fec500a9d70 2024-03-29T11:28:43.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.895+0000 7fec3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:43.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:43.895+0000 7fec3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:44.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.287+0000 7fec54b0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:44.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.287+0000 7fec54b0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:44.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.287+0000 7fec54b0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:28:44.277 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.287+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec5013c7b0 con 0x7fec500a9d70 2024-03-29T11:28:44.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.288+0000 7fec3effd640 1 -- 172.21.15.19:0/1661148420 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fec44052220 con 0x7fec500a9d70 2024-03-29T11:28:44.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.288+0000 7fec3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:44.278 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.288+0000 7fec3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:44.278 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:44.278 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:44.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.291+0000 7fec54b0e640 10 librados: watch_flush enter 2024-03-29T11:28:44.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 10 librados: watch_flush exit 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 >> v1:172.21.15.19:6816/3721232097 conn(0x7fec3006dea0 legacy=0x7fec30070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 10 monclient: shutdown 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 >> v1:172.21.15.62:6789/0 conn(0x7fec500a9d70 legacy=0x7fec50139a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 shutdown_connections 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 >> 172.21.15.19:0/1661148420 conn(0x7fec50066a00 msgr2=0x7fec50066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 shutdown_connections 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.292+0000 7fec54b0e640 1 -- 172.21.15.19:0/1661148420 wait complete. 2024-03-29T11:28:44.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:44.293+0000 7fec54b0e640 1 librados: shutdown 2024-03-29T11:28:44.298 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:51.105 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:51.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.263+0000 7f468d2ac640 10 monclient: get_monmap_and_config 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.263+0000 7f468d2ac640 10 monclient: build_initial_monmap 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.263+0000 7f468d2ac640 10 monclient: monmap: 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:51.264381+0000 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:51.264381+0000 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:51.254 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:51.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.267+0000 7f468d2ac640 1 Processor -- start 2024-03-29T11:28:51.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.267+0000 7f468d2ac640 1 -- start start 2024-03-29T11:28:51.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.267+0000 7f468d2ac640 10 monclient: init 2024-03-29T11:28:51.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:51.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:51.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient(hunting): picked mon.noname-c con 0x7f4680006d80 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:51.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient(hunting): picked mon.noname-b con 0x7f46800cc310 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:51.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient(hunting): picked mon.noname-a con 0x7f46800c9430 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:51.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4680008470 con 0x7f46800c9430 2024-03-29T11:28:51.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468002be30 con 0x7f4680006d80 2024-03-29T11:28:51.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468002c110 con 0x7f46800cc310 2024-03-29T11:28:51.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:51.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.268+0000 7f468d2ac640 10 monclient(hunting): authenticate will time out at 1451.531880s 2024-03-29T11:28:51.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.271+0000 7f46877fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f46800cc310 0x7f46800c8d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:51822/0 (socket says 172.21.15.19:51822) 2024-03-29T11:28:51.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f46877fe640 1 -- 172.21.15.19:0/2792780312 learned_addr learned my addr 172.21.15.19:0/2792780312 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:51.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299728758 0 0) 0x7f468002c110 con 0x7f46800cc310 2024-03-29T11:28:51.265 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4678003520 con 0x7f46800cc310 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946420619 0 0) 0x7f4678003520 con 0x7f46800cc310 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.272+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 >> v1:172.21.15.62:6789/0 conn(0x7f4680006d80 legacy=0x7f46800cbc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 >> v1:172.21.15.19:6789/0 conn(0x7f46800c9430 legacy=0x7f46800d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 1 monclient: found mon.noname-b 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468002c3f0 con 0x7f46800cc310 2024-03-29T11:28:51.266 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f467c001670 con 0x7f46800cc310 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.274+0000 7f4686ffd640 10 monclient: dump: 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:51.267 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.275+0000 7f468d2ac640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:28:51.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.275+0000 7f468d2ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.275+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f467c0018d0 con 0x7f46800cc310 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.275+0000 7f4686ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.275+0000 7f468d2ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.275+0000 7f468d2ac640 4 set_mon_vals no callback set 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.276+0000 7f468d2ac640 10 monclient: shutdown 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.276+0000 7f468d2ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.276+0000 7f468d2ac640 1 -- 172.21.15.19:0/2792780312 >> v1:172.21.15.19:6790/0 conn(0x7f46800cc310 legacy=0x7f46800c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.276+0000 7f4686ffd640 1 -- 172.21.15.19:0/2792780312 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f467c001ca0 con 0x7f46800cc310 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.276+0000 7f4686ffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:28:51.269 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.278+0000 7f468d2ac640 1 -- 172.21.15.19:0/2792780312 shutdown_connections 2024-03-29T11:28:51.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.278+0000 7f468d2ac640 1 -- 172.21.15.19:0/2792780312 >> 172.21.15.19:0/2792780312 conn(0x7f4680005c00 msgr2=0x7f4680006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:51.270 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.279+0000 7f468d2ac640 1 -- 172.21.15.19:0/2792780312 shutdown_connections 2024-03-29T11:28:51.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.279+0000 7f468d2ac640 1 -- 172.21.15.19:0/2792780312 wait complete. 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.280+0000 7f468d2ac640 10 monclient: build_initial_monmap 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.280+0000 7f468d2ac640 10 monclient: monmap: 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:51.273 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.280+0000 7f468d2ac640 1 librados: starting msgr at 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.280+0000 7f468d2ac640 1 librados: starting objecter 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.281+0000 7f468d2ac640 1 Processor -- start 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.281+0000 7f468d2ac640 1 -- start start 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.281+0000 7f468d2ac640 1 librados: setting wanted keys 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.281+0000 7f468d2ac640 1 librados: calling monclient init 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.281+0000 7f468d2ac640 10 monclient: init 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient(hunting): picked mon.noname-a con 0x7f4680006d80 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient(hunting): picked mon.noname-c con 0x7f46800c9430 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient(hunting): picked mon.noname-b con 0x7f46800cc310 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468002d940 con 0x7f4680006d80 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468002f950 con 0x7f46800cc310 2024-03-29T11:28:51.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468002fc30 con 0x7f46800c9430 2024-03-29T11:28:51.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:51.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.282+0000 7f468d2ac640 10 monclient(hunting): authenticate will time out at 1451.545780s 2024-03-29T11:28:51.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f46877fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f46800c9430 0x7f46801557f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:46666/0 (socket says 172.21.15.19:46666) 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f46877fe640 1 -- 172.21.15.19:0/2567668147 learned_addr learned my addr 172.21.15.19:0/2567668147 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 10 client.?.objecter ms_handle_connect 0x7f46800c9430 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098948037 0 0) 0x7f468002fc30 con 0x7f46800c9430 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.288+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c004eb0 con 0x7f46800c9430 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1722320753 0 0) 0x7f466c004eb0 con 0x7f46800c9430 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f468002fc30 con 0x7f46800c9430 2024-03-29T11:28:51.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f467c001ca0 con 0x7f46800c9430 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:51.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:51.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 192935101 0 0) 0x7f468002fc30 con 0x7f46800c9430 2024-03-29T11:28:51.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 >> v1:172.21.15.62:6789/0 conn(0x7f46800cc310 legacy=0x7f4680155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 >> v1:172.21.15.19:6789/0 conn(0x7f4680006d80 legacy=0x7f468002e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 monclient: found mon.c 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468002d5d0 con 0x7f46800c9430 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 10 monclient: _renew_subs 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468002ea80 con 0x7f46800c9430 2024-03-29T11:28:51.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 10 monclient: _renew_subs 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4680175120 con 0x7f46800c9430 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f468d2ac640 1 librados: init done 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f467c004b10 con 0x7f46800c9430 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4684ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.289+0000 7f4685ffb640 4 set_mon_vals no callback set 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.290+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f467c001ca0 con 0x7f46800c9430 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.290+0000 7f4684ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.290+0000 7f4684ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.290+0000 7f4684ff9640 10 monclient: dump: 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:51.287 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.291+0000 7f468d2ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.291+0000 7f468d2ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.291+0000 7f468d2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:51.288 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.291+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4668005080 con 0x7f46800c9430 2024-03-29T11:28:51.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.297+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f467c01aaf0 con 0x7f46800c9430 2024-03-29T11:28:51.289 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.297+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f467c054550 con 0x7f46800c9430 2024-03-29T11:28:51.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.298+0000 7f4687fff640 10 monclient: get_auth_request con 0x7f466c06dfa0 auth_method 0 2024-03-29T11:28:51.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 10 client.4604.objecter ms_dispatch 0x7f4688000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:51.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:51.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:51.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:28:51.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 10 client.4604.objecter ms_handle_connect 0x7f466c06dfa0 2024-03-29T11:28:51.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f467c0507f0 con 0x7f46800c9430 2024-03-29T11:28:51.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:51.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.303+0000 7f4684ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:51.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.637+0000 7f468d2ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:51.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.637+0000 7f468d2ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:51.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.637+0000 7f468d2ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:28:51.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.637+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4668005330 con 0x7f46800c9430 2024-03-29T11:28:51.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.640+0000 7f4684ff9640 1 -- 172.21.15.19:0/2567668147 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f467c050990 con 0x7f46800c9430 2024-03-29T11:28:51.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.640+0000 7f4684ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:51.631 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.640+0000 7f4684ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:51.631 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:51.631 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:51.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.644+0000 7f468d2ac640 10 librados: watch_flush enter 2024-03-29T11:28:51.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.644+0000 7f468d2ac640 10 librados: watch_flush exit 2024-03-29T11:28:51.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.644+0000 7f468d2ac640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:28:51.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.644+0000 7f468d2ac640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:28:51.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.645+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 >> v1:172.21.15.19:6816/3721232097 conn(0x7f466c06dfa0 legacy=0x7f466c070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.645+0000 7f468d2ac640 10 monclient: shutdown 2024-03-29T11:28:51.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.645+0000 7f468d2ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:51.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.645+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 >> v1:172.21.15.19:6790/0 conn(0x7f46800c9430 legacy=0x7f46801557f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:51.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.645+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 shutdown_connections 2024-03-29T11:28:51.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.645+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 >> 172.21.15.19:0/2567668147 conn(0x7f4680005c00 msgr2=0x7f4680005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:51.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.648+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 shutdown_connections 2024-03-29T11:28:51.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.649+0000 7f468d2ac640 1 -- 172.21.15.19:0/2567668147 wait complete. 2024-03-29T11:28:51.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:51.649+0000 7f468d2ac640 1 librados: shutdown 2024-03-29T11:28:51.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:58.463 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:58.610 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.620+0000 7ff444ac2640 10 monclient: get_monmap_and_config 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.620+0000 7ff444ac2640 10 monclient: build_initial_monmap 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.620+0000 7ff444ac2640 10 monclient: monmap: 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:28:58.621245+0000 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:28:58.621245+0000 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:28:58.611 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:58.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.622+0000 7ff444ac2640 1 Processor -- start 2024-03-29T11:28:58.612 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.622+0000 7ff444ac2640 1 -- start start 2024-03-29T11:28:58.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.622+0000 7ff444ac2640 10 monclient: init 2024-03-29T11:28:58.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:58.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:58.613 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient(hunting): picked mon.noname-b con 0x7ff438006eb0 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient(hunting): picked mon.noname-c con 0x7ff438038f10 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient(hunting): picked mon.noname-a con 0x7ff438036690 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff438033c90 con 0x7ff438036690 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4380389f0 con 0x7ff438038f10 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff438030890 con 0x7ff438006eb0 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.623+0000 7ff444ac2640 10 monclient(hunting): authenticate will time out at 1458.886652s 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.624+0000 7ff43f7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7ff438006eb0 0x7ff438031760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:54802/0 (socket says 172.21.15.19:54802) 2024-03-29T11:28:58.614 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.624+0000 7ff43f7fe640 1 -- 172.21.15.19:0/848650504 learned_addr learned my addr 172.21.15.19:0/848650504 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:58.615 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.625+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767593901 0 0) 0x7ff438030890 con 0x7ff438006eb0 2024-03-29T11:28:58.615 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.625+0000 7ff43e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:58.615 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.625+0000 7ff43e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:58.615 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.625+0000 7ff43e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:58.615 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.625+0000 7ff43e7fc640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:28:58.616 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.625+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff430003520 con 0x7ff438006eb0 2024-03-29T11:28:58.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1422945461 0 0) 0x7ff430003520 con 0x7ff438006eb0 2024-03-29T11:28:58.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 >> v1:172.21.15.62:6789/0 conn(0x7ff438038f10 legacy=0x7ff438039330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 >> v1:172.21.15.19:6789/0 conn(0x7ff438036690 legacy=0x7ff438036b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:58.622 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380cbc40 con 0x7ff438006eb0 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff434001690 con 0x7ff438006eb0 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: dump: 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:58.623 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff444ac2640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff444ac2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff434001860 con 0x7ff438006eb0 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 1 -- 172.21.15.19:0/848650504 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff434001c70 con 0x7ff438006eb0 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:58.624 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff43e7fc640 10 monclient: dump: 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff444ac2640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:58.625 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.629+0000 7ff444ac2640 4 set_mon_vals no callback set 2024-03-29T11:28:58.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.631+0000 7ff444ac2640 10 monclient: shutdown 2024-03-29T11:28:58.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.631+0000 7ff444ac2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:58.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.631+0000 7ff444ac2640 1 -- 172.21.15.19:0/848650504 >> v1:172.21.15.19:6790/0 conn(0x7ff438006eb0 legacy=0x7ff438031760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.635+0000 7ff444ac2640 1 -- 172.21.15.19:0/848650504 shutdown_connections 2024-03-29T11:28:58.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.635+0000 7ff444ac2640 1 -- 172.21.15.19:0/848650504 >> 172.21.15.19:0/848650504 conn(0x7ff438005c20 msgr2=0x7ff438006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:58.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.635+0000 7ff444ac2640 1 -- 172.21.15.19:0/848650504 shutdown_connections 2024-03-29T11:28:58.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.638+0000 7ff444ac2640 1 -- 172.21.15.19:0/848650504 wait complete. 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.639+0000 7ff444ac2640 10 monclient: build_initial_monmap 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.639+0000 7ff444ac2640 10 monclient: monmap: 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:58.632 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 1 librados: starting msgr at 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 1 librados: starting objecter 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 1 Processor -- start 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 1 -- start start 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 1 librados: setting wanted keys 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 1 librados: calling monclient init 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.640+0000 7ff444ac2640 10 monclient: init 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.641+0000 7ff444ac2640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.641+0000 7ff444ac2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 10 monclient(hunting): picked mon.noname-a con 0x7ff438006eb0 addr v1:172.21.15.19:6789/0 2024-03-29T11:28:58.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 10 monclient(hunting): picked mon.noname-b con 0x7ff438036690 addr v1:172.21.15.62:6789/0 2024-03-29T11:28:58.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 10 monclient(hunting): picked mon.noname-c con 0x7ff438038f10 addr v1:172.21.15.19:6790/0 2024-03-29T11:28:58.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43800b6a0 con 0x7ff438006eb0 2024-03-29T11:28:58.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43800a7e0 con 0x7ff438036690 2024-03-29T11:28:58.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43800b910 con 0x7ff438038f10 2024-03-29T11:28:58.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:58.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.642+0000 7ff444ac2640 10 monclient(hunting): authenticate will time out at 1458.905297s 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.644+0000 7ff43f7fe640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7ff438006eb0 0x7ff438009b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:47135/0 (socket says 172.21.15.19:47135) 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.644+0000 7ff43f7fe640 1 -- 172.21.15.19:0/3662424787 learned_addr learned my addr 172.21.15.19:0/3662424787 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.644+0000 7ff42bfff640 10 client.?.objecter ms_handle_connect 0x7ff438006eb0 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.644+0000 7ff42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760290906 0 0) 0x7ff43800b6a0 con 0x7ff438006eb0 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:58.636 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424004eb0 con 0x7ff438006eb0 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3213141328 0 0) 0x7ff424004eb0 con 0x7ff438006eb0 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff43800b6a0 con 0x7ff438006eb0 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff434001b70 con 0x7ff438006eb0 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): dump: 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:58.637 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:58.638 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:58.638 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:58.638 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:58.638 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:58.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4182784916 0 0) 0x7ff43800b6a0 con 0x7ff438006eb0 2024-03-29T11:28:58.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 >> v1:172.21.15.19:6790/0 conn(0x7ff438038f10 legacy=0x7ff4380c84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.639 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 >> v1:172.21.15.62:6789/0 conn(0x7ff438036690 legacy=0x7ff4380c7da0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 monclient: found mon.a 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43800a260 con 0x7ff438006eb0 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient: _finish_auth 0 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff42bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:58.640 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff444ac2640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff444ac2640 10 monclient: _renew_subs 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff444ac2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.645+0000 7ff444ac2640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43800a510 con 0x7ff438006eb0 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.646+0000 7ff444ac2640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.646+0000 7ff444ac2640 10 monclient: _renew_subs 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.646+0000 7ff444ac2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.646+0000 7ff444ac2640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff438186b00 con 0x7ff438006eb0 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.646+0000 7ff444ac2640 1 librados: init done 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff434004bf0 con 0x7ff438006eb0 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff42bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7ff434004f00 con 0x7ff438006eb0 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff42bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff42bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:58.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff42bfff640 10 monclient: dump: 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff429ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff429ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff429ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:58.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff420005080 con 0x7ff438006eb0 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.647+0000 7ff43d7fa640 4 set_mon_vals no callback set 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.648+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7ff4340050a0 con 0x7ff438006eb0 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.648+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7ff434084ca0 con 0x7ff438006eb0 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.649+0000 7ff42bfff640 10 client.4557.objecter ms_dispatch 0x7ff440000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.649+0000 7ff42bfff640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.649+0000 7ff42bfff640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.649+0000 7ff42bfff640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.649+0000 7ff43effd640 10 monclient: get_auth_request con 0x7ff42406db90 auth_method 0 2024-03-29T11:28:58.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.649+0000 7ff42bfff640 10 client.4557.objecter ms_handle_connect 0x7ff42406db90 2024-03-29T11:28:58.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.656+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff434050580 con 0x7ff438006eb0 2024-03-29T11:28:58.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.656+0000 7ff42bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:58.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.656+0000 7ff42bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.983+0000 7ff429ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.983+0000 7ff429ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.983+0000 7ff429ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.983+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff420005330 con 0x7ff438006eb0 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.984+0000 7ff42bfff640 1 -- 172.21.15.19:0/3662424787 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7ff4340524b0 con 0x7ff438006eb0 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.984+0000 7ff42bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:58.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.984+0000 7ff42bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:58.977 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:28:58.977 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:58.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.988+0000 7ff429ffb640 10 librados: watch_flush enter 2024-03-29T11:28:58.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 10 librados: watch_flush exit 2024-03-29T11:28:58.979 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T11:28:58.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 10 client.4557.objecter successfully canceled tick 2024-03-29T11:28:58.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 >> v1:172.21.15.19:6816/3721232097 conn(0x7ff42406db90 legacy=0x7ff424070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 10 monclient: shutdown 2024-03-29T11:28:58.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:58.982 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.989+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 >> v1:172.21.15.19:6789/0 conn(0x7ff438006eb0 legacy=0x7ff438009b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:58.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.992+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 shutdown_connections 2024-03-29T11:28:58.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.992+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 >> 172.21.15.19:0/3662424787 conn(0x7ff438005c20 msgr2=0x7ff438031520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:58.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.993+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 shutdown_connections 2024-03-29T11:28:58.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.993+0000 7ff429ffb640 1 -- 172.21.15.19:0/3662424787 wait complete. 2024-03-29T11:28:58.983 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:28:58.993+0000 7ff429ffb640 1 librados: shutdown 2024-03-29T11:28:59.001 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:05.808 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:05.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.958+0000 7f35f9e2e640 10 monclient: get_monmap_and_config 2024-03-29T11:29:05.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.958+0000 7f35f9e2e640 10 monclient: build_initial_monmap 2024-03-29T11:29:05.949 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.958+0000 7f35f9e2e640 10 monclient: monmap: 2024-03-29T11:29:05.949 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:05.949 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:05.949 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:05.959092+0000 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:05.959092+0000 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:05.950 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:05.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.959+0000 7f35f9e2e640 1 Processor -- start 2024-03-29T11:29:05.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.959+0000 7f35f9e2e640 1 -- start start 2024-03-29T11:29:05.953 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.959+0000 7f35f9e2e640 10 monclient: init 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f35f40b0770 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f35f40ad850 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f35f413c1b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f40ac8e0 con 0x7f35f40ad850 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f40ad330 con 0x7f35f413c1b0 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f3fff640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f35f413c1b0 0x7f35f414a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:36706/0 (socket says 172.21.15.19:36706) 2024-03-29T11:29:05.954 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f3fff640 1 -- 172.21.15.19:0/2316748648 learned_addr learned my addr 172.21.15.19:0/2316748648 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2316748648 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f40acfe0 con 0x7f35f40b0770 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f9e2e640 10 monclient(hunting): authenticate will time out at 1466.223685s 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206618539 0 0) 0x7f35f40ad330 con 0x7f35f413c1b0 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35dc003520 con 0x7f35f413c1b0 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088939917 0 0) 0x7f35f40acfe0 con 0x7f35f40b0770 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:05.955 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:05.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:05.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.960+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35f40ad330 con 0x7f35f40b0770 2024-03-29T11:29:05.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.961+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1751153680 0 0) 0x7f35dc003520 con 0x7f35f413c1b0 2024-03-29T11:29:05.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.961+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 >> v1:172.21.15.19:6790/0 conn(0x7f35f40b0770 legacy=0x7f35f4072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.956 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.961+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 >> v1:172.21.15.19:6789/0 conn(0x7f35f40ad850 legacy=0x7f35f40aaae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:05.958 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 1 monclient: found mon.noname-c 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.62:6789/0 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f40ab1f0 con 0x7f35f413c1b0 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f35ec001940 con 0x7f35f413c1b0 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f27fc640 10 monclient: dump: 2024-03-29T11:29:05.959 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f9e2e640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.964+0000 7f35f9e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 <== mon.1 v1:172.21.15.62:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35ec001aa0 con 0x7f35f413c1b0 2024-03-29T11:29:05.960 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f27fc640 1 -- 172.21.15.19:0/2316748648 <== mon.1 v1:172.21.15.62:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f35ec001eb0 con 0x7f35f413c1b0 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f27fc640 10 monclient: dump: 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:05.961 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f9e2e640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.965+0000 7f35f9e2e640 4 set_mon_vals no callback set 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.966+0000 7f35f9e2e640 10 monclient: shutdown 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.966+0000 7f35f9e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.966+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2316748648 >> v1:172.21.15.62:6789/0 conn(0x7f35f413c1b0 legacy=0x7f35f414a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.966+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2316748648 shutdown_connections 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.967+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2316748648 >> 172.21.15.19:0/2316748648 conn(0x7f35f406b660 msgr2=0x7f35f406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.967+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2316748648 shutdown_connections 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.967+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2316748648 wait complete. 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.967+0000 7f35f9e2e640 10 monclient: build_initial_monmap 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.967+0000 7f35f9e2e640 10 monclient: monmap: 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:05.962 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 1 librados: starting msgr at 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 1 librados: starting objecter 2024-03-29T11:29:05.963 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 1 Processor -- start 2024-03-29T11:29:05.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 1 -- start start 2024-03-29T11:29:05.969 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 1 librados: setting wanted keys 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 1 librados: calling monclient init 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.968+0000 7f35f9e2e640 10 monclient: init 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.969+0000 7f35f9e2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.969+0000 7f35f9e2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.969+0000 7f35f9e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f35f40ad850 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.969+0000 7f35f9e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f35f40b0770 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.969+0000 7f35f9e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f35f413c1b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.969+0000 7f35f9e2e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f413e320 con 0x7f35f40b0770 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35f9e2e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f413cd00 con 0x7f35f40ad850 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35f3fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f35f413c1b0 0x7f35f41d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:39130/0 (socket says 172.21.15.19:39130) 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35f3fff640 1 -- 172.21.15.19:0/2803086959 learned_addr learned my addr 172.21.15.19:0/2803086959 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:05.970 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35dbfff640 10 client.?.objecter ms_handle_connect 0x7f35f413c1b0 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f4135f30 con 0x7f35f413c1b0 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35f9e2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35f9e2e640 10 monclient(hunting): authenticate will time out at 1466.233752s 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.970+0000 7f35dbfff640 10 client.?.objecter ms_handle_connect 0x7f35f40ad850 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502349716 0 0) 0x7f35f4135f30 con 0x7f35f413c1b0 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e0004eb0 con 0x7f35f413c1b0 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1154815244 0 0) 0x7f35e0004eb0 con 0x7f35f413c1b0 2024-03-29T11:29:05.971 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35f4135f30 con 0x7f35f413c1b0 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f35ec001ef0 con 0x7f35f413c1b0 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): dump: 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:05.972 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.971+0000 7f35dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4075048339 0 0) 0x7f35f4135f30 con 0x7f35f413c1b0 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 >> v1:172.21.15.62:6789/0 conn(0x7f35f40ad850 legacy=0x7f35f413db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 >> v1:172.21.15.19:6789/0 conn(0x7f35f40b0770 legacy=0x7f35f4135820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 1 monclient: found mon.c 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f41360c0 con 0x7f35f413c1b0 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.972+0000 7f35dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:29:05.973 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 10 monclient: _renew_subs 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f413d240 con 0x7f35f413c1b0 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 10 monclient: _renew_subs 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f413d720 con 0x7f35f413c1b0 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f9e2e640 1 librados: init done 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35ec004e10 con 0x7f35f413c1b0 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.973+0000 7f35f1ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.974+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f35ec005120 con 0x7f35f413c1b0 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.974+0000 7f35dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.974+0000 7f35dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:05.974 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.974+0000 7f35dbfff640 10 monclient: dump: 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.975+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f35ec01b0f0 con 0x7f35f413c1b0 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f35ec084ed0 con 0x7f35f413c1b0 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35dbfff640 10 client.4613.objecter ms_dispatch 0x7f35f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:05.975 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35dbfff640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35dbfff640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35dbfff640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35f9e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35f9e2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35f9e2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d4005080 con 0x7f35f413c1b0 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35f37fe640 10 monclient: get_auth_request con 0x7f35e006dc20 auth_method 0 2024-03-29T11:29:05.976 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.976+0000 7f35dbfff640 10 client.4613.objecter ms_handle_connect 0x7f35e006dc20 2024-03-29T11:29:05.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.983+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f35ec0507b0 con 0x7f35f413c1b0 2024-03-29T11:29:05.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.983+0000 7f35dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:05.977 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:05.983+0000 7f35dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:06.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.306+0000 7f35f9e2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:06.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.306+0000 7f35f9e2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:06.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.306+0000 7f35f9e2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:06.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.306+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35d4005330 con 0x7f35f413c1b0 2024-03-29T11:29:06.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.307+0000 7f35dbfff640 1 -- 172.21.15.19:0/2803086959 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f35ec0526e0 con 0x7f35f413c1b0 2024-03-29T11:29:06.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.307+0000 7f35dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:06.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.307+0000 7f35dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:06.298 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:06.298 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:06.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.310+0000 7f35f9e2e640 10 librados: watch_flush enter 2024-03-29T11:29:06.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.310+0000 7f35f9e2e640 10 librados: watch_flush exit 2024-03-29T11:29:06.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.310+0000 7f35f9e2e640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:29:06.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.310+0000 7f35f9e2e640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:29:06.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 >> v1:172.21.15.19:6816/3721232097 conn(0x7f35e006dc20 legacy=0x7f35e00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:06.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 10 monclient: shutdown 2024-03-29T11:29:06.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:06.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 >> v1:172.21.15.19:6790/0 conn(0x7f35f413c1b0 legacy=0x7f35f41d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:06.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 shutdown_connections 2024-03-29T11:29:06.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 >> 172.21.15.19:0/2803086959 conn(0x7f35f406b660 msgr2=0x7f35f4138c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:06.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 shutdown_connections 2024-03-29T11:29:06.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 -- 172.21.15.19:0/2803086959 wait complete. 2024-03-29T11:29:06.303 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:06.311+0000 7f35f9e2e640 1 librados: shutdown 2024-03-29T11:29:06.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:13.125 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:13.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.281+0000 7fc218dcf640 10 monclient: get_monmap_and_config 2024-03-29T11:29:13.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.281+0000 7fc218dcf640 10 monclient: build_initial_monmap 2024-03-29T11:29:13.272 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.281+0000 7fc218dcf640 10 monclient: monmap: 2024-03-29T11:29:13.272 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:13.272 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.272 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:13.281969+0000 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:13.281969+0000 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:13.273 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.274 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.282+0000 7fc218dcf640 1 Processor -- start 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.282+0000 7fc218dcf640 1 -- start start 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.282+0000 7fc218dcf640 10 monclient: init 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc2141457f0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc21413f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc2140a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2140aaef0 con 0x7fc21413f650 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc214145bd0 con 0x7fc2140a9d70 2024-03-29T11:29:13.275 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc214145eb0 con 0x7fc2141457f0 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc218dcf640 10 monclient(hunting): authenticate will time out at 1473.546691s 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.283+0000 7fc211d9b640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc21413f650 0x7fc2140b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:47718/0 (socket says 172.21.15.19:47718) 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc211d9b640 1 -- 172.21.15.19:0/3062675137 learned_addr learned my addr 172.21.15.19:0/3062675137 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052861971 0 0) 0x7fc2140aaef0 con 0x7fc21413f650 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:29:13.276 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1fc003520 con 0x7fc21413f650 2024-03-29T11:29:13.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798518254 0 0) 0x7fc214145bd0 con 0x7fc2140a9d70 2024-03-29T11:29:13.279 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): _negotiate 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2140aaef0 con 0x7fc2140a9d70 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879959064 0 0) 0x7fc1fc003520 con 0x7fc21413f650 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 >> v1:172.21.15.19:6790/0 conn(0x7fc2141457f0 legacy=0x7fc21413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.284+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 >> v1:172.21.15.62:6789/0 conn(0x7fc2140a9d70 legacy=0x7fc21414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: _finish_hunting 0 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 1 monclient: found mon.noname-a 2024-03-29T11:29:13.280 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214146340 con 0x7fc21413f650 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: _finish_auth 0 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: _check_auth_tickets 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc208001690 con 0x7fc21413f650 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: dump: 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:13.281 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc218dcf640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc218dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2080017f0 con 0x7fc21413f650 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:13.282 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 1 -- 172.21.15.19:0/3062675137 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc208001c00 con 0x7fc21413f650 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc21159a640 10 monclient: dump: 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:13.283 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc218dcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.285+0000 7fc218dcf640 4 set_mon_vals no callback set 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.286+0000 7fc218dcf640 10 monclient: shutdown 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.286+0000 7fc218dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.286+0000 7fc218dcf640 1 -- 172.21.15.19:0/3062675137 >> v1:172.21.15.19:6789/0 conn(0x7fc21413f650 legacy=0x7fc2140b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.286+0000 7fc218dcf640 1 -- 172.21.15.19:0/3062675137 shutdown_connections 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.286+0000 7fc218dcf640 1 -- 172.21.15.19:0/3062675137 >> 172.21.15.19:0/3062675137 conn(0x7fc214066a00 msgr2=0x7fc214066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.286+0000 7fc218dcf640 1 -- 172.21.15.19:0/3062675137 shutdown_connections 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.287+0000 7fc218dcf640 1 -- 172.21.15.19:0/3062675137 wait complete. 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 10 monclient: build_initial_monmap 2024-03-29T11:29:13.284 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 10 monclient: monmap: 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.285 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 1 librados: starting msgr at 2024-03-29T11:29:13.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 1 librados: starting objecter 2024-03-29T11:29:13.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 1 Processor -- start 2024-03-29T11:29:13.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 1 -- start start 2024-03-29T11:29:13.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 1 librados: setting wanted keys 2024-03-29T11:29:13.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 1 librados: calling monclient init 2024-03-29T11:29:13.286 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.288+0000 7fc218dcf640 10 monclient: init 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient(hunting): picked mon.noname-b con 0x7fc2140a9d70 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient(hunting): picked mon.noname-c con 0x7fc21413f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient(hunting): picked mon.noname-a con 0x7fc2141457f0 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21413fda0 con 0x7fc2141457f0 2024-03-29T11:29:13.291 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc214141b90 con 0x7fc2140a9d70 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc214141de0 con 0x7fc21413f650 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.289+0000 7fc218dcf640 10 monclient(hunting): authenticate will time out at 1473.552762s 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc212d9d640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7fc2141457f0 0x7fc2141e6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:48136/0 (socket says 172.21.15.19:48136) 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc212d9d640 1 -- 172.21.15.19:0/772973833 learned_addr learned my addr 172.21.15.19:0/772973833 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 client.?.objecter ms_handle_connect 0x7fc2141457f0 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 client.?.objecter ms_handle_connect 0x7fc2140a9d70 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424377573 0 0) 0x7fc214141b90 con 0x7fc2140a9d70 2024-03-29T11:29:13.292 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.290+0000 7fc202ffd640 10 monclient(hunting): my global_id is 4555 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f4004eb0 con 0x7fc2140a9d70 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618060803 0 0) 0x7fc21413fda0 con 0x7fc2141457f0 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc214141b90 con 0x7fc2141457f0 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 client.?.objecter ms_handle_connect 0x7fc21413f650 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:13.293 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647051598 0 0) 0x7fc214141de0 con 0x7fc21413f650 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc21413fda0 con 0x7fc21413f650 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1158420041 0 0) 0x7fc214141b90 con 0x7fc2141457f0 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc214141de0 con 0x7fc2141457f0 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573820649 0 0) 0x7fc1f4004eb0 con 0x7fc2140a9d70 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc214141b90 con 0x7fc2140a9d70 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.291+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc20c001cf0 con 0x7fc2141457f0 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:13.294 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:13.295 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc2040017a0 con 0x7fc2140a9d70 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:13.296 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3615532383 0 0) 0x7fc21413fda0 con 0x7fc21413f650 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1f4004eb0 con 0x7fc21413f650 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc208001b00 con 0x7fc21413f650 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:13.297 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1606305647 0 0) 0x7fc214141de0 con 0x7fc2141457f0 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 >> v1:172.21.15.19:6790/0 conn(0x7fc21413f650 legacy=0x7fc2141e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 >> v1:172.21.15.62:6789/0 conn(0x7fc2140a9d70 legacy=0x7fc2141424e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 monclient: found mon.a 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:13.298 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214140da0 con 0x7fc2141457f0 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 10 monclient: _renew_subs 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc214071fc0 con 0x7fc2141457f0 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 10 monclient: _renew_subs 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:13.299 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc214142dd0 con 0x7fc2141457f0 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.292+0000 7fc218dcf640 1 librados: init done 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc20c001fd0 con 0x7fc2141457f0 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc202ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc210d99640 4 set_mon_vals no callback set 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc20c001e50 con 0x7fc2141457f0 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc202ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.293+0000 7fc202ffd640 10 monclient: dump: 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:13.300 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.294+0000 7fc218dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.294+0000 7fc218dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.294+0000 7fc218dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.294+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f8005080 con 0x7fc2141457f0 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.296+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fc20c003de0 con 0x7fc2141457f0 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc21259c640 10 monclient: get_auth_request con 0x7fc1f406e160 auth_method 0 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7fc20c084dc0 con 0x7fc2141457f0 2024-03-29T11:29:13.301 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc202ffd640 10 client.4563.objecter ms_dispatch 0x7fc214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc202ffd640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc202ffd640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc202ffd640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.298+0000 7fc202ffd640 10 client.4563.objecter ms_handle_connect 0x7fc1f406e160 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.304+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc20c0506a0 con 0x7fc2141457f0 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.304+0000 7fc202ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:13.302 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.304+0000 7fc202ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:13.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.679+0000 7fc218dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:13.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.679+0000 7fc218dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:13.670 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.679+0000 7fc218dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:13.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.679+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1f8005330 con 0x7fc2141457f0 2024-03-29T11:29:13.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.680+0000 7fc202ffd640 1 -- 172.21.15.19:0/772973833 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7fc20c0525d0 con 0x7fc2141457f0 2024-03-29T11:29:13.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.680+0000 7fc202ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:13.671 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.680+0000 7fc202ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:13.672 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:13.672 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 10 librados: watch_flush enter 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 10 librados: watch_flush exit 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 >> v1:172.21.15.19:6816/3721232097 conn(0x7fc1f406e160 legacy=0x7fc1f4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 10 monclient: shutdown 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 >> v1:172.21.15.19:6789/0 conn(0x7fc2141457f0 legacy=0x7fc2141e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 shutdown_connections 2024-03-29T11:29:13.675 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 >> 172.21.15.19:0/772973833 conn(0x7fc214066a00 msgr2=0x7fc2140b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:13.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.684+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 shutdown_connections 2024-03-29T11:29:13.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.685+0000 7fc218dcf640 1 -- 172.21.15.19:0/772973833 wait complete. 2024-03-29T11:29:13.676 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:13.685+0000 7fc218dcf640 1 librados: shutdown 2024-03-29T11:29:13.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:20.497 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:20.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.634+0000 7f7549377640 10 monclient: get_monmap_and_config 2024-03-29T11:29:20.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.634+0000 7f7549377640 10 monclient: build_initial_monmap 2024-03-29T11:29:20.626 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.634+0000 7f7549377640 10 monclient: monmap: 2024-03-29T11:29:20.626 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:20.626 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:20.626 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:20.635061+0000 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:20.635061+0000 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.635+0000 7f7549377640 1 Processor -- start 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.635+0000 7f7549377640 1 -- start start 2024-03-29T11:29:20.627 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.635+0000 7f7549377640 10 monclient: init 2024-03-29T11:29:20.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:20.628 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient(hunting): picked mon.noname-b con 0x7f75440afb20 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient(hunting): picked mon.noname-c con 0x7f75440ad7c0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient(hunting): picked mon.noname-a con 0x7f754413d720 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440aeb10 con 0x7f754413d720 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440aedf0 con 0x7f75440ad7c0 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440af0d0 con 0x7f75440afb20 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7549377640 10 monclient(hunting): authenticate will time out at 1480.899138s 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7542ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f75440afb20 0x7f754413ffb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:59822/0 (socket says 172.21.15.19:59822) 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7542ffd640 1 -- 172.21.15.19:0/2482102571 learned_addr learned my addr 172.21.15.19:0/2482102571 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844181014 0 0) 0x7f75440af0d0 con 0x7f75440afb20 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:20.629 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c003520 con 0x7f75440afb20 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399040495 0 0) 0x7f75440aedf0 con 0x7f75440ad7c0 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75440af0d0 con 0x7f75440ad7c0 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652437806 0 0) 0x7f752c003520 con 0x7f75440afb20 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 >> v1:172.21.15.62:6789/0 conn(0x7f75440ad7c0 legacy=0x7f754413d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.630 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.636+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 >> v1:172.21.15.19:6789/0 conn(0x7f754413d720 legacy=0x7f754414b8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 1 monclient: found mon.noname-b 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440af3b0 con 0x7f75440afb20 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: _finish_auth 0 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7538001690 con 0x7f75440afb20 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7549377640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7549377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:20.632 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: dump: 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7549377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7538001870 con 0x7f75440afb20 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:20.633 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 1 -- 172.21.15.19:0/2482102571 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7538001c80 con 0x7f75440afb20 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7549377640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7549377640 4 set_mon_vals no callback set 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.639+0000 7f7541ffb640 10 monclient: dump: 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:20.634 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 10 monclient: shutdown 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 1 -- 172.21.15.19:0/2482102571 >> v1:172.21.15.19:6790/0 conn(0x7f75440afb20 legacy=0x7f754413ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 1 -- 172.21.15.19:0/2482102571 shutdown_connections 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 1 -- 172.21.15.19:0/2482102571 >> 172.21.15.19:0/2482102571 conn(0x7f75440b35e0 msgr2=0x7f75440b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 1 -- 172.21.15.19:0/2482102571 shutdown_connections 2024-03-29T11:29:20.635 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.640+0000 7f7549377640 1 -- 172.21.15.19:0/2482102571 wait complete. 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.641+0000 7f7549377640 10 monclient: build_initial_monmap 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.641+0000 7f7549377640 10 monclient: monmap: 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:20.641 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.641+0000 7f7549377640 1 librados: starting msgr at 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.641+0000 7f7549377640 1 librados: starting objecter 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.642+0000 7f7549377640 1 Processor -- start 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.642+0000 7f7549377640 1 -- start start 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.642+0000 7f7549377640 1 librados: setting wanted keys 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.642+0000 7f7549377640 1 librados: calling monclient init 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.642+0000 7f7549377640 10 monclient: init 2024-03-29T11:29:20.642 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient(hunting): picked mon.noname-c con 0x7f75440ad7c0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7542ffd640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f75440ad7c0 0x7f75441d6f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60498/0 (socket says 172.21.15.19:60498) 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7542ffd640 1 -- 172.21.15.19:0/3419519993 learned_addr learned my addr 172.21.15.19:0/3419519993 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient(hunting): picked mon.noname-a con 0x7f75440afb20 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient(hunting): picked mon.noname-b con 0x7f754413d720 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7544141330 con 0x7f75440afb20 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 10 client.?.objecter ms_handle_connect 0x7f75440ad7c0 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7544135740 con 0x7f754413d720 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7544141aa0 con 0x7f75440ad7c0 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:20.643 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f7549377640 10 monclient(hunting): authenticate will time out at 1480.906462s 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060527084 0 0) 0x7f7544141aa0 con 0x7f75440ad7c0 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.643+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7530004eb0 con 0x7f75440ad7c0 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 963235771 0 0) 0x7f7530004eb0 con 0x7f75440ad7c0 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7544141aa0 con 0x7f75440ad7c0 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7538001c40 con 0x7f75440ad7c0 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 10 monclient(hunting): dump: 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:20.644 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1346636663 0 0) 0x7f7544141aa0 con 0x7f75440ad7c0 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 >> v1:172.21.15.62:6789/0 conn(0x7f754413d720 legacy=0x7f7544135030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.644+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 >> v1:172.21.15.19:6789/0 conn(0x7f75440afb20 legacy=0x7f7544134920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 1 monclient: found mon.c 2024-03-29T11:29:20.645 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7544142fc0 con 0x7f75440ad7c0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f752b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f7549377640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f7549377640 10 monclient: _renew_subs 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.646+0000 7f7549377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7544141d40 con 0x7f75440ad7c0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7538004bf0 con 0x7f75440ad7c0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f752b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f7538004f00 con 0x7f75440ad7c0 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f752b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:20.646 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f752b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f752b7fe640 10 monclient: dump: 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f7549377640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f7549377640 10 monclient: _renew_subs 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f7549377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:20.647 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75440b31c0 con 0x7f75440ad7c0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f7549377640 1 librados: init done 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.647+0000 7f75417fa640 4 set_mon_vals no callback set 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.648+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f753801aed0 con 0x7f75440ad7c0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.648+0000 7f7549377640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.648+0000 7f7549377640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.648+0000 7f7549377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75441418f0 con 0x7f75440ad7c0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3482790580 0 0) 0x7f7538084d20 con 0x7f75440ad7c0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f752b7fe640 10 client.4625.objecter ms_dispatch 0x7f7544000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f752b7fe640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f752b7fe640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f752b7fe640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.649+0000 7f75427fc640 10 monclient: get_auth_request con 0x7f753006dbf0 auth_method 0 2024-03-29T11:29:20.648 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.650+0000 7f752b7fe640 10 client.4625.objecter ms_handle_connect 0x7f753006dbf0 2024-03-29T11:29:20.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.657+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7538050660 con 0x7f75440ad7c0 2024-03-29T11:29:20.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.657+0000 7f752b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:20.649 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.657+0000 7f752b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:20.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.933+0000 7f7549377640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:20.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.933+0000 7f7549377640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:20.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.933+0000 7f7549377640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:20.924 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.933+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7544075f10 con 0x7f75440ad7c0 2024-03-29T11:29:20.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.935+0000 7f752b7fe640 1 -- 172.21.15.19:0/3419519993 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2274845622) 0x7f7538001210 con 0x7f75440ad7c0 2024-03-29T11:29:20.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.935+0000 7f752b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:20.926 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.935+0000 7f752b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:20.926 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:20.926 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":10,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:19:24.065435+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:24.065435+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:20.929 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 10 librados: watch_flush enter 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 10 librados: watch_flush exit 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 10 client.4625.objecter successfully canceled tick 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 >> v1:172.21.15.19:6816/3721232097 conn(0x7f753006dbf0 legacy=0x7f75300700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 10 monclient: shutdown 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 >> v1:172.21.15.19:6790/0 conn(0x7f75440ad7c0 legacy=0x7f75441d6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 shutdown_connections 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 >> 172.21.15.19:0/3419519993 conn(0x7f75440b35e0 msgr2=0x7f75440a9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 shutdown_connections 2024-03-29T11:29:20.930 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 -- 172.21.15.19:0/3419519993 wait complete. 2024-03-29T11:29:20.931 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:20.938+0000 7f7549377640 1 librados: shutdown 2024-03-29T11:29:20.944 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:27.750 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.884+0000 7f909e64a640 10 monclient: get_monmap_and_config 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.884+0000 7f909e64a640 10 monclient: build_initial_monmap 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.884+0000 7f909e64a640 10 monclient: monmap: 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:27.885264+0000 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:27.885264+0000 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:27.876 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:27.877 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:27.877 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.878 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.885+0000 7f909e64a640 1 Processor -- start 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.885+0000 7f909e64a640 1 -- start start 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.885+0000 7f909e64a640 10 monclient: init 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient(hunting): picked mon.noname-a con 0x7f90980ae250 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient(hunting): picked mon.noname-b con 0x7f90980a9720 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f90977fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90980a9720 0x7f909813f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:55738/0 (socket says 172.21.15.19:55738) 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f90977fe640 1 -- 172.21.15.19:0/1139732616 learned_addr learned my addr 172.21.15.19:0/1139732616 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient(hunting): picked mon.noname-c con 0x7f909814bba0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90980a9b80 con 0x7f90980ae250 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90980aa7b0 con 0x7f909814bba0 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90980afbf0 con 0x7f90980a9720 2024-03-29T11:29:27.879 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f909e64a640 10 monclient(hunting): authenticate will time out at 1488.149619s 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805631280 0 0) 0x7f90980afbf0 con 0x7f90980a9720 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f9096ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f9096ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f9096ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f9096ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.886+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9080003520 con 0x7f90980a9720 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.887+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2247829721 0 0) 0x7f9080003520 con 0x7f90980a9720 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.887+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 >> v1:172.21.15.62:6789/0 conn(0x7f909814bba0 legacy=0x7f909814df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:27.880 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.887+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 >> v1:172.21.15.19:6789/0 conn(0x7f90980ae250 legacy=0x7f9098142b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 1 monclient: found mon.noname-b 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098143280 con 0x7f90980a9720 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f908c001690 con 0x7f90980a9720 2024-03-29T11:29:27.882 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: dump: 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f909e64a640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:29:27.883 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f909e64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f908c0017f0 con 0x7f90980a9720 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 1 -- 172.21.15.19:0/1139732616 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f908c001c40 con 0x7f90980a9720 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f909e64a640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f909e64a640 4 set_mon_vals no callback set 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.889+0000 7f9096ffd640 10 monclient: dump: 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.884 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.885 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.885 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:27.885 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:27.885 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:27.885 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.885 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 10 monclient: shutdown 2024-03-29T11:29:27.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:27.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 >> v1:172.21.15.19:6790/0 conn(0x7f90980a9720 legacy=0x7f909813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 shutdown_connections 2024-03-29T11:29:27.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 >> 172.21.15.19:0/1139732616 conn(0x7f90980b35e0 msgr2=0x7f90980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:27.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 shutdown_connections 2024-03-29T11:29:27.886 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.890+0000 7f909e64a640 1 -- 172.21.15.19:0/1139732616 wait complete. 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.891+0000 7f909e64a640 10 monclient: build_initial_monmap 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.891+0000 7f909e64a640 10 monclient: monmap: 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:27.891 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.892+0000 7f909e64a640 1 librados: starting msgr at 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.892+0000 7f909e64a640 1 librados: starting objecter 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.892+0000 7f909e64a640 1 Processor -- start 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.892+0000 7f909e64a640 1 -- start start 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.893+0000 7f909e64a640 1 librados: setting wanted keys 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.893+0000 7f909e64a640 1 librados: calling monclient init 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.893+0000 7f909e64a640 10 monclient: init 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.893+0000 7f909e64a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:27.892 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 10 monclient(hunting): picked mon.noname-c con 0x7f90980a9720 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 10 monclient(hunting): picked mon.noname-b con 0x7f90980ae250 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f9097fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90980a9720 0x7f909813e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:56344/0 (socket says 172.21.15.19:56344) 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f90977fe640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7f90980ae250 0x7f90981da870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:47180/0 (socket says 172.21.15.19:47180) 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 10 monclient(hunting): picked mon.noname-a con 0x7f909814bba0 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f9097fff640 1 -- 172.21.15.19:0/482291753 learned_addr learned my addr 172.21.15.19:0/482291753 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909813efb0 con 0x7f909814bba0 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909813f440 con 0x7f90980ae250 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f909813de00 con 0x7f90980a9720 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.894+0000 7f909e64a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f909e64a640 10 monclient(hunting): authenticate will time out at 1488.157993s 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 client.?.objecter ms_handle_connect 0x7f90980a9720 2024-03-29T11:29:27.893 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 client.?.objecter ms_handle_connect 0x7f90980ae250 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535644063 0 0) 0x7f909813f440 con 0x7f90980ae250 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:27.894 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9084004eb0 con 0x7f90980ae250 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973169997 0 0) 0x7f909813de00 con 0x7f90980a9720 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.895+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f909813f440 con 0x7f90980a9720 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 189747546 0 0) 0x7f9084004eb0 con 0x7f90980ae250 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f909813de00 con 0x7f90980ae250 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f908c001bc0 con 0x7f90980ae250 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:27.895 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:27.896 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:27.896 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.896 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.896 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3344276992 0 0) 0x7f909813f440 con 0x7f90980a9720 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9084004eb0 con 0x7f90980a9720 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f90880018a0 con 0x7f90980a9720 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.897 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:27.898 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.896+0000 7f9094ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:27.898 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.897+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 338957200 0 0) 0x7f9084004eb0 con 0x7f90980a9720 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.897+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 >> v1:172.21.15.62:6789/0 conn(0x7f90980ae250 legacy=0x7f90981da870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.897+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 >> v1:172.21.15.19:6789/0 conn(0x7f909814bba0 legacy=0x7f90981e1e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:27.899 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 1 monclient: found mon.c 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909813df90 con 0x7f90980a9720 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:27.900 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 10 monclient: _renew_subs 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909813b5d0 con 0x7f90980a9720 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90880035f0 con 0x7f90980a9720 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9095ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f9088003a00 con 0x7f90980a9720 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f9094ff9640 10 monclient: dump: 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:27.901 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 10 monclient: _renew_subs 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90981db160 con 0x7f90980a9720 2024-03-29T11:29:27.902 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.898+0000 7f909e64a640 1 librados: init done 2024-03-29T11:29:27.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.900+0000 7f909e64a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:27.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.900+0000 7f909e64a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:27.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.900+0000 7f909e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:27.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.900+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f909813b970 con 0x7f90980a9720 2024-03-29T11:29:27.903 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.900+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f9088003be0 con 0x7f90980a9720 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f90977fe640 10 monclient: get_auth_request con 0x7f908406de40 auth_method 0 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7f9088084980 con 0x7f90980a9720 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f9094ff9640 10 client.4631.objecter ms_dispatch 0x7f9098000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f9094ff9640 3 client.4631.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f9094ff9640 3 client.4631.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f9094ff9640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.901+0000 7f9094ff9640 10 client.4631.objecter ms_handle_connect 0x7f908406de40 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.907+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90880502c0 con 0x7f90980a9720 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.907+0000 7f9094ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:27.904 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:27.907+0000 7f9094ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:28.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.191+0000 7f909e64a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:28.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.191+0000 7f909e64a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:28.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.191+0000 7f909e64a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:28.182 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.191+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f909813fe80 con 0x7f90980a9720 2024-03-29T11:29:28.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.192+0000 7f9094ff9640 1 -- 172.21.15.19:0/482291753 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7f9088084030 con 0x7f90980a9720 2024-03-29T11:29:28.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.192+0000 7f9094ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:28.183 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.192+0000 7f9094ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:28.183 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:28.184 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:28.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 10 librados: watch_flush enter 2024-03-29T11:29:28.186 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 10 librados: watch_flush exit 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 >> v1:172.21.15.19:6816/3721232097 conn(0x7f908406de40 legacy=0x7f9084070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 10 monclient: shutdown 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.194+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 >> v1:172.21.15.19:6790/0 conn(0x7f90980a9720 legacy=0x7f909813e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.195+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 shutdown_connections 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.195+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 >> 172.21.15.19:0/482291753 conn(0x7f90980b35e0 msgr2=0x7f90980b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.195+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 shutdown_connections 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.195+0000 7f909e64a640 1 -- 172.21.15.19:0/482291753 wait complete. 2024-03-29T11:29:28.187 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:28.195+0000 7f909e64a640 1 librados: shutdown 2024-03-29T11:29:28.200 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:35.008 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:35.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.157+0000 7f90b1c0f640 10 monclient: get_monmap_and_config 2024-03-29T11:29:35.148 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.157+0000 7f90b1c0f640 10 monclient: build_initial_monmap 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.157+0000 7f90b1c0f640 10 monclient: monmap: 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:35.157854+0000 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:35.157854+0000 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:35.149 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 1 Processor -- start 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 1 -- start start 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 10 monclient: init 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f90ac0b0770 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f90ac14a700 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:35.151 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f90ac0ac5a0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:35.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.158+0000 7f90b1c0f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac0aba70 con 0x7f90ac14a700 2024-03-29T11:29:35.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.159+0000 7f90b1c0f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac0aecb0 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.159+0000 7f90b1c0f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac0af150 con 0x7f90ac0b0770 2024-03-29T11:29:35.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.159+0000 7f90b1c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:35.152 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.159+0000 7f90b1c0f640 10 monclient(hunting): authenticate will time out at 1495.422014s 2024-03-29T11:29:35.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.160+0000 7f90ab7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90ac0b0770 0x7f90ac149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:39038/0 (socket says 172.21.15.19:39038) 2024-03-29T11:29:35.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.160+0000 7f90ab7fe640 1 -- 172.21.15.19:0/4054262684 learned_addr learned my addr 172.21.15.19:0/4054262684 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:35.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.161+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493056425 0 0) 0x7f90ac0af150 con 0x7f90ac0b0770 2024-03-29T11:29:35.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.161+0000 7f90aa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:35.153 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.161+0000 7f90aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:35.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.161+0000 7f90aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:35.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.161+0000 7f90aa7fc640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:29:35.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.161+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9094003520 con 0x7f90ac0b0770 2024-03-29T11:29:35.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.162+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3139685682 0 0) 0x7f9094003520 con 0x7f90ac0b0770 2024-03-29T11:29:35.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.162+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 >> v1:172.21.15.62:6789/0 conn(0x7f90ac0ac5a0 legacy=0x7f90ac151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:35.154 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.162+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 >> v1:172.21.15.19:6789/0 conn(0x7f90ac14a700 legacy=0x7f90ac0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.163+0000 7f90aa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.163+0000 7f90aa7fc640 1 monclient: found mon.noname-b 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.163+0000 7f90aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.163+0000 7f90aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.163+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac0ac9d0 con 0x7f90ac0b0770 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90b1c0f640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90b1c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:35.156 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f909c001670 con 0x7f90ac0b0770 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 10 monclient: dump: 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:35.157 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:35.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90b1c0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:35.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f909c001980 con 0x7f90ac0b0770 2024-03-29T11:29:35.158 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90b1c0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.164+0000 7f90b1c0f640 4 set_mon_vals no callback set 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90aa7fc640 1 -- 172.21.15.19:0/4054262684 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f909c001d50 con 0x7f90ac0b0770 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90aa7fc640 10 monclient: dump: 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:35.160 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90b1c0f640 10 monclient: shutdown 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90b1c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.165+0000 7f90b1c0f640 1 -- 172.21.15.19:0/4054262684 >> v1:172.21.15.19:6790/0 conn(0x7f90ac0b0770 legacy=0x7f90ac149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.166+0000 7f90b1c0f640 1 -- 172.21.15.19:0/4054262684 shutdown_connections 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.166+0000 7f90b1c0f640 1 -- 172.21.15.19:0/4054262684 >> 172.21.15.19:0/4054262684 conn(0x7f90ac06b660 msgr2=0x7f90ac06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.166+0000 7f90b1c0f640 1 -- 172.21.15.19:0/4054262684 shutdown_connections 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.167+0000 7f90b1c0f640 1 -- 172.21.15.19:0/4054262684 wait complete. 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 10 monclient: build_initial_monmap 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 10 monclient: monmap: 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:35.161 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 1 librados: starting msgr at 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 1 librados: starting objecter 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 1 Processor -- start 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 1 -- start start 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 1 librados: setting wanted keys 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 1 librados: calling monclient init 2024-03-29T11:29:35.162 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.168+0000 7f90b1c0f640 10 monclient: init 2024-03-29T11:29:35.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:35.165 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient(hunting): picked mon.noname-c con 0x7f90ac0ac5a0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient(hunting): picked mon.noname-b con 0x7f90ac0b0770 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient(hunting): picked mon.noname-a con 0x7f90ac14a700 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac125e70 con 0x7f90ac14a700 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac127020 con 0x7f90ac0b0770 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90ac127290 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.169+0000 7f90b1c0f640 10 monclient(hunting): authenticate will time out at 1495.432859s 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.170+0000 7f90ab7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f90ac0ac5a0 0x7f90ac1deb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:39712/0 (socket says 172.21.15.19:39712) 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.170+0000 7f90ab7fe640 1 -- 172.21.15.19:0/2781982688 learned_addr learned my addr 172.21.15.19:0/2781982688 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.170+0000 7f9093fff640 10 client.?.objecter ms_handle_connect 0x7f90ac0ac5a0 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.170+0000 7f9093fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:35.166 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.171+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723646980 0 0) 0x7f90ac127290 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.171+0000 7f9093fff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.171+0000 7f9093fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.171+0000 7f9093fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.171+0000 7f9093fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.171+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9098004eb0 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26841493 0 0) 0x7f9098004eb0 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9098005300 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f909c001630 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 10 monclient(hunting): dump: 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.167 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 769620722 0 0) 0x7f9098005300 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.168 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.172+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 >> v1:172.21.15.62:6789/0 conn(0x7f90ac0b0770 legacy=0x7f90ac143fa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:35.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 >> v1:172.21.15.19:6789/0 conn(0x7f90ac14a700 legacy=0x7f90ac1e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:35.169 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 1 monclient: found mon.c 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ac124f10 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 10 monclient: _finish_auth 0 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 10 client.?.objecter ms_handle_connect 0x7f90ac14a700 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f9093fff640 10 client.4637.objecter resend_mon_ops 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 10 monclient: _renew_subs 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:35.170 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ac144b90 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 10 monclient: _renew_subs 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ac126300 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.175+0000 7f90b1c0f640 1 librados: init done 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.177+0000 7f90b1c0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.177+0000 7f90b1c0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.177+0000 7f90b1c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.177+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ac125c50 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f909c004cb0 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f9093fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f909c001630 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f9093fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f9093fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:35.171 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f9093fff640 10 monclient: dump: 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:35.172 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.178+0000 7f90a9ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.181+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f909c01ac90 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7f909c084a30 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f9093fff640 10 client.4637.objecter ms_dispatch 0x7f90ac000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f9093fff640 3 client.4637.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f9093fff640 3 client.4637.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f9093fff640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:29:35.173 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f90aaffd640 10 monclient: get_auth_request con 0x7f909806dff0 auth_method 0 2024-03-29T11:29:35.174 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.182+0000 7f9093fff640 10 client.4637.objecter ms_handle_connect 0x7f909806dff0 2024-03-29T11:29:35.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.187+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f909c050310 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.178 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.187+0000 7f9093fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:35.179 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.187+0000 7f9093fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:35.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.498+0000 7f90b1c0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:35.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.498+0000 7f90b1c0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:35.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.498+0000 7f90b1c0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:35.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.498+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90ac125c50 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.501+0000 7f9093fff640 1 -- 172.21.15.19:0/2781982688 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7f909c050310 con 0x7f90ac0ac5a0 2024-03-29T11:29:35.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.501+0000 7f9093fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:35.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.501+0000 7f9093fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:35.493 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:35.493 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:35.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.505+0000 7f90b1c0f640 10 librados: watch_flush enter 2024-03-29T11:29:35.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.505+0000 7f90b1c0f640 10 librados: watch_flush exit 2024-03-29T11:29:35.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.505+0000 7f90b1c0f640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:29:35.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.505+0000 7f90b1c0f640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:29:35.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 >> v1:172.21.15.19:6816/3721232097 conn(0x7f909806dff0 legacy=0x7f90980704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:35.497 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 10 monclient: shutdown 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 >> v1:172.21.15.19:6790/0 conn(0x7f90ac0ac5a0 legacy=0x7f90ac1deb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 shutdown_connections 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 >> 172.21.15.19:0/2781982688 conn(0x7f90ac06b660 msgr2=0x7f90ac0b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 shutdown_connections 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 -- 172.21.15.19:0/2781982688 wait complete. 2024-03-29T11:29:35.498 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:35.506+0000 7f90b1c0f640 1 librados: shutdown 2024-03-29T11:29:35.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:42.323 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:42.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.485+0000 7fe986a3e640 10 monclient: get_monmap_and_config 2024-03-29T11:29:42.476 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.485+0000 7fe986a3e640 10 monclient: build_initial_monmap 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.485+0000 7fe986a3e640 10 monclient: monmap: 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:42.485839+0000 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:42.485839+0000 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:42.477 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:42.479 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.487+0000 7fe986a3e640 1 Processor -- start 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.490+0000 7fe986a3e640 1 -- start start 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.490+0000 7fe986a3e640 10 monclient: init 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fe9800b1f60 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fe980071eb0 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fe980143d10 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98006ef80 con 0x7fe980071eb0 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98006f260 con 0x7fe980143d10 2024-03-29T11:29:42.482 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9800b40c0 con 0x7fe9800b1f60 2024-03-29T11:29:42.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:42.483 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.491+0000 7fe986a3e640 10 monclient(hunting): authenticate will time out at 1502.754141s 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.493+0000 7fe97ffff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe9800b1f60 0x7fe980143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:50186/0 (socket says 172.21.15.19:50186) 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.493+0000 7fe97ffff640 1 -- 172.21.15.19:0/2034896451 learned_addr learned my addr 172.21.15.19:0/2034896451 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545849738 0 0) 0x7fe9800b40c0 con 0x7fe9800b1f60 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe968003520 con 0x7fe9800b1f60 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831021330 0 0) 0x7fe98006ef80 con 0x7fe980071eb0 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:42.491 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9800b40c0 con 0x7fe980071eb0 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035481823 0 0) 0x7fe968003520 con 0x7fe9800b1f60 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 >> v1:172.21.15.62:6789/0 conn(0x7fe980143d10 legacy=0x7fe980146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 >> v1:172.21.15.19:6789/0 conn(0x7fe980071eb0 legacy=0x7fe980146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 monclient: found mon.noname-b 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980155d70 con 0x7fe9800b1f60 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient: _finish_auth 0 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:42.492 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe986a3e640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe986a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe970001690 con 0x7fe9800b1f60 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.495+0000 7fe97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe97effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe97effd640 10 monclient: dump: 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:42.493 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe986a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe970001870 con 0x7fe9800b1f60 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe97effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe986a3e640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.496+0000 7fe986a3e640 4 set_mon_vals no callback set 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe97effd640 1 -- 172.21.15.19:0/2034896451 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe970001690 con 0x7fe9800b1f60 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe97effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe97effd640 10 monclient: dump: 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:42.494 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe986a3e640 10 monclient: shutdown 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe986a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe986a3e640 1 -- 172.21.15.19:0/2034896451 >> v1:172.21.15.19:6790/0 conn(0x7fe9800b1f60 legacy=0x7fe980143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe986a3e640 1 -- 172.21.15.19:0/2034896451 shutdown_connections 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.497+0000 7fe986a3e640 1 -- 172.21.15.19:0/2034896451 >> 172.21.15.19:0/2034896451 conn(0x7fe980066a00 msgr2=0x7fe980066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.498+0000 7fe986a3e640 1 -- 172.21.15.19:0/2034896451 shutdown_connections 2024-03-29T11:29:42.495 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.498+0000 7fe986a3e640 1 -- 172.21.15.19:0/2034896451 wait complete. 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.499+0000 7fe986a3e640 10 monclient: build_initial_monmap 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.499+0000 7fe986a3e640 10 monclient: monmap: 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.499+0000 7fe986a3e640 1 librados: starting msgr at 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.499+0000 7fe986a3e640 1 librados: starting objecter 2024-03-29T11:29:42.496 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.499+0000 7fe986a3e640 1 Processor -- start 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.500+0000 7fe986a3e640 1 -- start start 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.500+0000 7fe986a3e640 1 librados: setting wanted keys 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.500+0000 7fe986a3e640 1 librados: calling monclient init 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.500+0000 7fe986a3e640 10 monclient: init 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.500+0000 7fe986a3e640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fe9800b1f60 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fe980143d10 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:42.499 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fe9800714b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9801554c0 con 0x7fe9800714b0 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe980153c40 con 0x7fe9800b1f60 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9801557a0 con 0x7fe980143d10 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.501+0000 7fe986a3e640 10 monclient(hunting): authenticate will time out at 1502.764085s 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.502+0000 7fe97f7fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fe980143d10 0x7fe980153530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:52844/0 (socket says 172.21.15.19:52844) 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.502+0000 7fe97f7fe640 1 -- 172.21.15.19:0/3635954998 learned_addr learned my addr 172.21.15.19:0/3635954998 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.502+0000 7fe97cff9640 10 client.?.objecter ms_handle_connect 0x7fe980143d10 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.502+0000 7fe97cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2250372827 0 0) 0x7fe9801557a0 con 0x7fe980143d10 2024-03-29T11:29:42.500 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe96c004eb0 con 0x7fe980143d10 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 620990314 0 0) 0x7fe96c004eb0 con 0x7fe980143d10 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9801557a0 con 0x7fe980143d10 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe9740018f0 con 0x7fe980143d10 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:42.501 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): dump: 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2300881420 0 0) 0x7fe9801557a0 con 0x7fe980143d10 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.503+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 >> v1:172.21.15.62:6789/0 conn(0x7fe9800b1f60 legacy=0x7fe980142800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:42.502 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 >> v1:172.21.15.19:6789/0 conn(0x7fe9800714b0 legacy=0x7fe9801e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 1 monclient: found mon.c 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980134780 con 0x7fe980143d10 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 client.?.objecter ms_handle_connect 0x7fe9800714b0 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:42.503 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe974001d20 con 0x7fe980143d10 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.506+0000 7fe97e7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fe974003a60 con 0x7fe980143d10 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe97cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe97cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe97cff9640 10 monclient: dump: 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:42.504 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 10 monclient: _renew_subs 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9801396b0 con 0x7fe980143d10 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 10 monclient: _renew_subs 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9801e83c0 con 0x7fe980143d10 2024-03-29T11:29:42.505 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.507+0000 7fe986a3e640 1 librados: init done 2024-03-29T11:29:42.506 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.508+0000 7fe986a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.508+0000 7fe986a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.508+0000 7fe986a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.508+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe98004efd0 con 0x7fe980143d10 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.512+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fe974003bc0 con 0x7fe980143d10 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.512+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7fe974084a90 con 0x7fe980143d10 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.513+0000 7fe97cff9640 10 client.4643.objecter ms_dispatch 0x7fe980000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.513+0000 7fe97cff9640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.513+0000 7fe97cff9640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.513+0000 7fe97ffff640 10 monclient: get_auth_request con 0x7fe96c06db30 auth_method 0 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.513+0000 7fe97cff9640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:29:42.507 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.513+0000 7fe97cff9640 10 client.4643.objecter ms_handle_connect 0x7fe96c06db30 2024-03-29T11:29:42.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.520+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9740501a0 con 0x7fe980143d10 2024-03-29T11:29:42.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.520+0000 7fe97cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:42.511 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.520+0000 7fe97cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:42.848 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.855+0000 7fe986a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:42.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.855+0000 7fe986a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:42.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.855+0000 7fe986a3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:42.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.855+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe980134910 con 0x7fe980143d10 2024-03-29T11:29:42.849 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.858+0000 7fe97cff9640 1 -- 172.21.15.19:0/3635954998 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7fe980134910 con 0x7fe980143d10 2024-03-29T11:29:42.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.858+0000 7fe97cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:42.850 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.858+0000 7fe97cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:42.851 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:42.851 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 10 librados: watch_flush enter 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 10 librados: watch_flush exit 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 >> v1:172.21.15.19:6816/3721232097 conn(0x7fe96c06db30 legacy=0x7fe96c06fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 10 monclient: shutdown 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:42.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.862+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 >> v1:172.21.15.19:6790/0 conn(0x7fe980143d10 legacy=0x7fe980153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:42.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.864+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 shutdown_connections 2024-03-29T11:29:42.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.864+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 >> 172.21.15.19:0/3635954998 conn(0x7fe980066a00 msgr2=0x7fe980066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:42.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.868+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 shutdown_connections 2024-03-29T11:29:42.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.868+0000 7fe986a3e640 1 -- 172.21.15.19:0/3635954998 wait complete. 2024-03-29T11:29:42.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:42.868+0000 7fe986a3e640 1 librados: shutdown 2024-03-29T11:29:42.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:49.688 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:49.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.843+0000 7f1ef9ed8640 10 monclient: get_monmap_and_config 2024-03-29T11:29:49.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.843+0000 7f1ef9ed8640 10 monclient: build_initial_monmap 2024-03-29T11:29:49.835 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.843+0000 7f1ef9ed8640 10 monclient: monmap: 2024-03-29T11:29:49.835 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:49.835 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:49.835 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:49.844628+0000 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:49.844628+0000 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.845+0000 7f1ef9ed8640 1 Processor -- start 2024-03-29T11:29:49.836 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.845+0000 7f1ef9ed8640 1 -- start start 2024-03-29T11:29:49.837 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.845+0000 7f1ef9ed8640 10 monclient: init 2024-03-29T11:29:49.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:49.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:49.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient(hunting): picked mon.noname-a con 0x7f1eec038c60 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:49.840 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient(hunting): picked mon.noname-b con 0x7f1eec0324b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient(hunting): picked mon.noname-c con 0x7f1eec0d11e0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0caf80 con 0x7f1eec038c60 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec039850 con 0x7f1eec0d11e0 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0c78d0 con 0x7f1eec0324b0 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef9ed8640 10 monclient(hunting): authenticate will time out at 1510.109367s 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef3fff640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1eec0324b0 0x7f1eec0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:59328/0 (socket says 172.21.15.19:59328) 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.846+0000 7f1ef3fff640 1 -- 172.21.15.19:0/743703120 learned_addr learned my addr 172.21.15.19:0/743703120 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893832481 0 0) 0x7f1eec0c78d0 con 0x7f1eec0324b0 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:29:49.841 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee4003520 con 0x7f1eec0324b0 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506952519 0 0) 0x7f1eec0caf80 con 0x7f1eec038c60 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.847+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec0c78d0 con 0x7f1eec038c60 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 890692328 0 0) 0x7f1ee4003520 con 0x7f1eec0324b0 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 >> v1:172.21.15.62:6789/0 conn(0x7f1eec0d11e0 legacy=0x7f1eec0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 >> v1:172.21.15.19:6789/0 conn(0x7f1eec038c60 legacy=0x7f1eec0c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 1 monclient: found mon.noname-b 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec0c7a60 con 0x7f1eec0324b0 2024-03-29T11:29:49.842 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1ee0001690 con 0x7f1eec0324b0 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef37fe640 10 monclient: dump: 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:49.843 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef9ed8640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.848+0000 7f1ef9ed8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ee0001860 con 0x7f1eec0324b0 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef37fe640 1 -- 172.21.15.19:0/743703120 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1ee0001c70 con 0x7f1eec0324b0 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef37fe640 10 monclient: dump: 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.844 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef9ed8640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:49.845 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef9ed8640 4 set_mon_vals no callback set 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.849+0000 7f1ef9ed8640 10 monclient: shutdown 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.850+0000 7f1ef9ed8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.850+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/743703120 >> v1:172.21.15.19:6790/0 conn(0x7f1eec0324b0 legacy=0x7f1eec0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.850+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/743703120 shutdown_connections 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.850+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/743703120 >> 172.21.15.19:0/743703120 conn(0x7f1eec005c20 msgr2=0x7f1eec006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.850+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/743703120 shutdown_connections 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.851+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/743703120 wait complete. 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 10 monclient: build_initial_monmap 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 10 monclient: monmap: 2024-03-29T11:29:49.854 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 1 librados: starting msgr at 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 1 librados: starting objecter 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 1 Processor -- start 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 1 -- start start 2024-03-29T11:29:49.855 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 1 librados: setting wanted keys 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 1 librados: calling monclient init 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.852+0000 7f1ef9ed8640 10 monclient: init 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.853+0000 7f1ef9ed8640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.853+0000 7f1ef9ed8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 10 monclient(hunting): picked mon.noname-b con 0x7f1eec0324b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 10 monclient(hunting): picked mon.noname-a con 0x7f1eec038c60 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 10 monclient(hunting): picked mon.noname-c con 0x7f1eec0d11e0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec035dc0 con 0x7f1eec038c60 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec036840 con 0x7f1eec0324b0 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0358a0 con 0x7f1eec0d11e0 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef9ed8640 10 monclient(hunting): authenticate will time out at 1510.117273s 2024-03-29T11:29:49.856 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef96d7640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f1eec0d11e0 0x7f1eec16b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:60244/0 (socket says 172.21.15.19:60244) 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef96d7640 1 -- 172.21.15.19:0/153372270 learned_addr learned my addr 172.21.15.19:0/153372270 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.854+0000 7f1ef17fa640 10 client.?.objecter ms_handle_connect 0x7f1eec0d11e0 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 client.?.objecter ms_handle_connect 0x7f1eec038c60 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564145730 0 0) 0x7f1eec0358a0 con 0x7f1eec0d11e0 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed8004eb0 con 0x7f1eec0d11e0 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109329646 0 0) 0x7f1eec035dc0 con 0x7f1eec038c60 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:49.857 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.855+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eec0358a0 con 0x7f1eec038c60 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 741666813 0 0) 0x7f1ed8004eb0 con 0x7f1eec0d11e0 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eec035dc0 con 0x7f1eec0d11e0 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1ee80018b0 con 0x7f1eec0d11e0 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): dump: 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.858 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3868427567 0 0) 0x7f1eec0358a0 con 0x7f1eec038c60 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ed8004eb0 con 0x7f1eec038c60 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1ee0001b70 con 0x7f1eec038c60 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 10 monclient(hunting): dump: 2024-03-29T11:29:49.859 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.856+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3084535743 0 0) 0x7f1ed8004eb0 con 0x7f1eec038c60 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 >> v1:172.21.15.19:6790/0 conn(0x7f1eec0d11e0 legacy=0x7f1eec16b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 >> v1:172.21.15.62:6789/0 conn(0x7f1eec0324b0 legacy=0x7f1eec02d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:49.860 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 1 monclient: found mon.a 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec035a30 con 0x7f1eec038c60 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 5 monclient: authenticate success, global_id 4578 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 10 monclient: _renew_subs 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec035270 con 0x7f1eec038c60 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 10 monclient: _renew_subs 2024-03-29T11:29:49.861 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec037020 con 0x7f1eec038c60 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef9ed8640 1 librados: init done 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ee0001cd0 con 0x7f1eec038c60 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f1ee0004c30 con 0x7f1eec038c60 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef17fa640 10 monclient: dump: 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:49.862 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.857+0000 7f1ef2ffd640 4 set_mon_vals no callback set 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.858+0000 7f1ef9ed8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.858+0000 7f1ef9ed8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.858+0000 7f1ef9ed8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.858+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ed4005080 con 0x7f1eec038c60 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.860+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f1ee001ac00 con 0x7f1eec038c60 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7f1ee0084950 con 0x7f1eec038c60 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef17fa640 10 client.4578.objecter ms_dispatch 0x7f1ef4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef17fa640 3 client.4578.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:49.863 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef17fa640 3 client.4578.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:49.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef17fa640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-29T11:29:49.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef8ed6640 10 monclient: get_auth_request con 0x7f1ed806de60 auth_method 0 2024-03-29T11:29:49.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.861+0000 7f1ef17fa640 10 client.4578.objecter ms_handle_connect 0x7f1ed806de60 2024-03-29T11:29:49.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.870+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ee0050230 con 0x7f1eec038c60 2024-03-29T11:29:49.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.870+0000 7f1ef17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:49.864 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:49.870+0000 7f1ef17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:50.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.265+0000 7f1ef9ed8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:50.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.265+0000 7f1ef9ed8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:50.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.265+0000 7f1ef9ed8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:29:50.257 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.265+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ed4005330 con 0x7f1eec038c60 2024-03-29T11:29:50.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.266+0000 7f1ef17fa640 1 -- 172.21.15.19:0/153372270 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7f1ee0052160 con 0x7f1eec038c60 2024-03-29T11:29:50.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.266+0000 7f1ef17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:50.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.266+0000 7f1ef17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:50.258 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:50.258 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 10 librados: watch_flush enter 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 10 librados: watch_flush exit 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 10 client.4578.objecter successfully canceled tick 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 >> v1:172.21.15.19:6816/3721232097 conn(0x7f1ed806de60 legacy=0x7f1ed8070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 10 monclient: shutdown 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:50.262 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.270+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 >> v1:172.21.15.19:6789/0 conn(0x7f1eec038c60 legacy=0x7f1eec0c6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:50.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.271+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 shutdown_connections 2024-03-29T11:29:50.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.271+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 >> 172.21.15.19:0/153372270 conn(0x7f1eec005c20 msgr2=0x7f1eec03bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:50.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.271+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 shutdown_connections 2024-03-29T11:29:50.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.271+0000 7f1ef9ed8640 1 -- 172.21.15.19:0/153372270 wait complete. 2024-03-29T11:29:50.263 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:50.272+0000 7f1ef9ed8640 1 librados: shutdown 2024-03-29T11:29:50.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:57.083 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.239+0000 7f030e84a640 10 monclient: get_monmap_and_config 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.239+0000 7f030e84a640 10 monclient: build_initial_monmap 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.239+0000 7f030e84a640 10 monclient: monmap: 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:29:57.241359+0000 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:29:57.241359+0000 2024-03-29T11:29:57.232 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.240+0000 7f030e84a640 1 Processor -- start 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.240+0000 7f030e84a640 1 -- start start 2024-03-29T11:29:57.233 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.240+0000 7f030e84a640 10 monclient: init 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient(hunting): picked mon.noname-a con 0x7f03080b4330 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient(hunting): picked mon.noname-b con 0x7f03081470b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient(hunting): picked mon.noname-c con 0x7f03081439b0 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f030806f2a0 con 0x7f03080b4330 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0308146b90 con 0x7f03081439b0 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080b2ca0 con 0x7f03081470b0 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:57.235 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.241+0000 7f030e84a640 10 monclient(hunting): authenticate will time out at 1517.505590s 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f03077fe640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f03081470b0 0x7f03080b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:44268/0 (socket says 172.21.15.19:44268) 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f03077fe640 1 -- 172.21.15.19:0/3884697928 learned_addr learned my addr 172.21.15.19:0/3884697928 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446995707 0 0) 0x7f03080b2ca0 con 0x7f03081470b0 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f0306ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f0306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f0306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f0306ffd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:29:57.236 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.243+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f0003520 con 0x7f03081470b0 2024-03-29T11:29:57.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.244+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233443436 0 0) 0x7f02f0003520 con 0x7f03081470b0 2024-03-29T11:29:57.237 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.244+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 >> v1:172.21.15.62:6789/0 conn(0x7f03081439b0 legacy=0x7f0308151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.247+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 >> v1:172.21.15.19:6789/0 conn(0x7f03080b4330 legacy=0x7f03080b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.247+0000 7f0306ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.247+0000 7f0306ffd640 1 monclient: found mon.noname-b 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.247+0000 7f0306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.247+0000 7f0306ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030806e0c0 con 0x7f03081470b0 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:57.241 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0300001a10 con 0x7f03081470b0 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f0306ffd640 10 monclient: dump: 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:57.242 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:57.243 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:57.243 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:57.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f030e84a640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:29:57.243 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.248+0000 7f030e84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.249+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0300001ca0 con 0x7f03081470b0 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.249+0000 7f0306ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.249+0000 7f030e84a640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.249+0000 7f030e84a640 4 set_mon_vals no callback set 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f0306ffd640 1 -- 172.21.15.19:0/3884697928 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f0300002030 con 0x7f03081470b0 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f0306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f0306ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f0306ffd640 10 monclient: dump: 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.244 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 10 monclient: shutdown 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 1 -- 172.21.15.19:0/3884697928 >> v1:172.21.15.19:6790/0 conn(0x7f03081470b0 legacy=0x7f03080b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 1 -- 172.21.15.19:0/3884697928 shutdown_connections 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 1 -- 172.21.15.19:0/3884697928 >> 172.21.15.19:0/3884697928 conn(0x7f0308066a00 msgr2=0x7f0308066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:57.245 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 1 -- 172.21.15.19:0/3884697928 shutdown_connections 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.250+0000 7f030e84a640 1 -- 172.21.15.19:0/3884697928 wait complete. 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.251+0000 7f030e84a640 10 monclient: build_initial_monmap 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.251+0000 7f030e84a640 10 monclient: monmap: 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:57.246 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 1 librados: starting msgr at 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 1 librados: starting objecter 2024-03-29T11:29:57.247 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 1 Processor -- start 2024-03-29T11:29:57.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 1 -- start start 2024-03-29T11:29:57.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 1 librados: setting wanted keys 2024-03-29T11:29:57.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 1 librados: calling monclient init 2024-03-29T11:29:57.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.252+0000 7f030e84a640 10 monclient: init 2024-03-29T11:29:57.252 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient(hunting): picked mon.noname-b con 0x7f03080b4330 addr v1:172.21.15.62:6789/0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient(hunting): picked mon.noname-a con 0x7f03081439b0 addr v1:172.21.15.19:6789/0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient(hunting): picked mon.noname-c con 0x7f03081470b0 addr v1:172.21.15.19:6790/0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080ad080 con 0x7f03081439b0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080abb70 con 0x7f03080b4330 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03080ab100 con 0x7f03081470b0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.253+0000 7f030e84a640 10 monclient(hunting): authenticate will time out at 1517.517521s 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.257+0000 7f030cde7640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7f03081470b0 0x7f0308142fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:50604/0 (socket says 172.21.15.19:50604) 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.257+0000 7f030cde7640 1 -- 172.21.15.19:0/4162560939 learned_addr learned my addr 172.21.15.19:0/4162560939 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 10 client.?.objecter ms_handle_connect 0x7f03081470b0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892038103 0 0) 0x7f03080ab100 con 0x7f03081470b0 2024-03-29T11:29:57.253 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.258+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f4004eb0 con 0x7f03081470b0 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357657631 0 0) 0x7f02f4004eb0 con 0x7f03081470b0 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03080ab100 con 0x7f03081470b0 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f02fc001750 con 0x7f03081470b0 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 10 monclient(hunting): dump: 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:57.254 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2965936211 0 0) 0x7f03080ab100 con 0x7f03081470b0 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 >> v1:172.21.15.62:6789/0 conn(0x7f03080b4330 legacy=0x7f03081e6f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.259+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 >> v1:172.21.15.19:6789/0 conn(0x7f03081439b0 legacy=0x7f03081428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 1 monclient: found mon.c 2024-03-29T11:29:57.255 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:57.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:57.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03080ab290 con 0x7f03081470b0 2024-03-29T11:29:57.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:57.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:57.256 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:57.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 5 monclient: authenticate success, global_id 4655 2024-03-29T11:29:57.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 10 monclient: _renew_subs 2024-03-29T11:29:57.258 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03080ab550 con 0x7f03081470b0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 10 monclient: _renew_subs 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0308145680 con 0x7f03081470b0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f030e84a640 1 librados: init done 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 10 client.4655.objecter ms_handle_connect 0x7f03081439b0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.260+0000 7f0304ff9640 10 client.4655.objecter resend_mon_ops 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02fc001d20 con 0x7f03081470b0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0304ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f02fc004a00 con 0x7f03081470b0 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0304ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0304ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:57.259 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0304ff9640 10 monclient: dump: 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.263+0000 7f0305ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.264+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f02fc01aa50 con 0x7f03081470b0 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.264+0000 7f0307fff640 10 monclient: get_auth_request con 0x7f02f406dc20 auth_method 0 2024-03-29T11:29:57.260 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.265+0000 7f0304ff9640 10 client.4655.objecter ms_handle_connect 0x7f02f406dc20 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.265+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7f02fc084a50 con 0x7f03081470b0 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.265+0000 7f0304ff9640 10 client.4655.objecter ms_dispatch 0x7f0308000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.265+0000 7f0304ff9640 3 client.4655.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.265+0000 7f0304ff9640 3 client.4655.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.265+0000 7f0304ff9640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.266+0000 7f030e84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.266+0000 7f030e84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.266+0000 7f030e84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:57.261 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.266+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8005080 con 0x7f03081470b0 2024-03-29T11:29:57.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.275+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02fc050330 con 0x7f03081470b0 2024-03-29T11:29:57.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.275+0000 7f0304ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:57.268 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.275+0000 7f0304ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:57.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.696+0000 7f030e84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:57.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.696+0000 7f030e84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:57.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.696+0000 7f030e84a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:29:57.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.696+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02e8005330 con 0x7f03081470b0 2024-03-29T11:29:57.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.698+0000 7f0304ff9640 1 -- 172.21.15.19:0/4162560939 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7f02fc052260 con 0x7f03081470b0 2024-03-29T11:29:57.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.698+0000 7f0304ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:57.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.698+0000 7f0304ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:57.691 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:29:57.692 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:57.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.702+0000 7f030e84a640 10 librados: watch_flush enter 2024-03-29T11:29:57.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.702+0000 7f030e84a640 10 librados: watch_flush exit 2024-03-29T11:29:57.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.702+0000 7f030e84a640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-29T11:29:57.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.702+0000 7f030e84a640 10 client.4655.objecter successfully canceled tick 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 >> v1:172.21.15.19:6816/3721232097 conn(0x7f02f406dc20 legacy=0x7f02f40700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 10 monclient: shutdown 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 >> v1:172.21.15.19:6790/0 conn(0x7f03081470b0 legacy=0x7f0308142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 shutdown_connections 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 >> 172.21.15.19:0/4162560939 conn(0x7f0308066a00 msgr2=0x7f03081d14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:57.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 shutdown_connections 2024-03-29T11:29:57.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 -- 172.21.15.19:0/4162560939 wait complete. 2024-03-29T11:29:57.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:29:57.703+0000 7f030e84a640 1 librados: shutdown 2024-03-29T11:29:57.712 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:04.521 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.683+0000 7fa42a87b640 10 monclient: get_monmap_and_config 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.683+0000 7fa42a87b640 10 monclient: build_initial_monmap 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.683+0000 7fa42a87b640 10 monclient: monmap: 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:30:04.685635+0000 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:30:04.685635+0000 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:04.677 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.685+0000 7fa42a87b640 1 Processor -- start 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.685+0000 7fa42a87b640 1 -- start start 2024-03-29T11:30:04.678 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.685+0000 7fa42a87b640 10 monclient: init 2024-03-29T11:30:04.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.686+0000 7fa42a87b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:04.679 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.686+0000 7fa42a87b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:04.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4240a9750 addr v1:172.21.15.19:6789/0 2024-03-29T11:30:04.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4240aad20 addr v1:172.21.15.62:6789/0 2024-03-29T11:30:04.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4240a9b30 addr v1:172.21.15.19:6790/0 2024-03-29T11:30:04.682 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4240af4d0 con 0x7fa4240a9750 2024-03-29T11:30:04.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4240af7b0 con 0x7fa4240aad20 2024-03-29T11:30:04.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4240acab0 con 0x7fa4240a9b30 2024-03-29T11:30:04.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:04.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a87b640 10 monclient(hunting): authenticate will time out at 1524.953107s 2024-03-29T11:30:04.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a07a640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa4240a9b30 0x7fa424149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:33410/0 (socket says 172.21.15.19:33410) 2024-03-29T11:30:04.683 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.689+0000 7fa42a07a640 1 -- 172.21.15.19:0/692257566 learned_addr learned my addr 172.21.15.19:0/692257566 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061215417 0 0) 0x7fa4240acab0 con 0x7fa4240a9b30 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 10 monclient(hunting): _negotiate 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa410003520 con 0x7fa4240a9b30 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700316116 0 0) 0x7fa4240af4d0 con 0x7fa4240a9750 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 10 monclient(hunting): _negotiate 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.691+0000 7fa428877640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:04.687 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4240acab0 con 0x7fa4240a9750 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787080033 0 0) 0x7fa4240af7b0 con 0x7fa4240aad20 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 10 monclient(hunting): _negotiate 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4240af4d0 con 0x7fa4240aad20 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513789568 0 0) 0x7fa4240acab0 con 0x7fa4240a9750 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 >> v1:172.21.15.19:6790/0 conn(0x7fa4240a9b30 legacy=0x7fa424149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.692+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 >> v1:172.21.15.62:6789/0 conn(0x7fa4240aad20 legacy=0x7fa424141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: _finish_hunting 0 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 1 monclient: found mon.noname-a 2024-03-29T11:30:04.688 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4240aba30 con 0x7fa4240a9750 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: _finish_auth 0 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: _check_auth_tickets 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa420001670 con 0x7fa4240a9750 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa42a87b640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa42a87b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 10 monclient: dump: 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:04.689 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa42a87b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.693+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa420001890 con 0x7fa4240a9750 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa428877640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa428877640 1 -- 172.21.15.19:0/692257566 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa420001ba0 con 0x7fa4240a9750 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa42a87b640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa42a87b640 4 set_mon_vals no callback set 2024-03-29T11:30:04.690 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa428877640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa428877640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa428877640 10 monclient: dump: 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:04.691 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa42a87b640 10 monclient: shutdown 2024-03-29T11:30:04.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa42a87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:04.693 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.694+0000 7fa42a87b640 1 -- 172.21.15.19:0/692257566 >> v1:172.21.15.19:6789/0 conn(0x7fa4240a9750 legacy=0x7fa4240ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.696+0000 7fa42a87b640 1 -- 172.21.15.19:0/692257566 shutdown_connections 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.696+0000 7fa42a87b640 1 -- 172.21.15.19:0/692257566 >> 172.21.15.19:0/692257566 conn(0x7fa4240673e0 msgr2=0x7fa4240b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.696+0000 7fa42a87b640 1 -- 172.21.15.19:0/692257566 shutdown_connections 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.696+0000 7fa42a87b640 1 -- 172.21.15.19:0/692257566 wait complete. 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.697+0000 7fa42a87b640 10 monclient: build_initial_monmap 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.697+0000 7fa42a87b640 10 monclient: monmap: 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.694 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.698+0000 7fa42a87b640 1 librados: starting msgr at 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.698+0000 7fa42a87b640 1 librados: starting objecter 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.698+0000 7fa42a87b640 1 Processor -- start 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.698+0000 7fa42a87b640 1 -- start start 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.698+0000 7fa42a87b640 1 librados: setting wanted keys 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.698+0000 7fa42a87b640 1 librados: calling monclient init 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.699+0000 7fa42a87b640 10 monclient: init 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.699+0000 7fa42a87b640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.700+0000 7fa42a87b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:04.695 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.700+0000 7fa42a87b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4240a9750 addr v1:172.21.15.19:6790/0 2024-03-29T11:30:04.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.700+0000 7fa42a87b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4240a9b30 addr v1:172.21.15.62:6789/0 2024-03-29T11:30:04.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.702+0000 7fa42a87b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4240aad20 addr v1:172.21.15.19:6789/0 2024-03-29T11:30:04.696 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.702+0000 7fa42a87b640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4241dbfc0 con 0x7fa4240aad20 2024-03-29T11:30:04.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.702+0000 7fa42a87b640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa424071960 con 0x7fa4240a9b30 2024-03-29T11:30:04.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.702+0000 7fa42a87b640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4241dc1f0 con 0x7fa4240a9750 2024-03-29T11:30:04.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.702+0000 7fa42a87b640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:04.697 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.702+0000 7fa42a87b640 10 monclient(hunting): authenticate will time out at 1524.966932s 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.703+0000 7fa429879640 1 --1- >> v1:172.21.15.19:6790/0 conn(0x7fa4240a9750 0x7fa4241db8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6790/0 says I am v1:172.21.15.19:41314/0 (socket says 172.21.15.19:41314) 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.703+0000 7fa429879640 1 -- 172.21.15.19:0/3529016279 learned_addr learned my addr 172.21.15.19:0/3529016279 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 10 client.?.objecter ms_handle_connect 0x7fa4240a9750 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805394385 0 0) 0x7fa4241dc1f0 con 0x7fa4240a9750 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:04.700 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.704+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa408004eb0 con 0x7fa4240a9750 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1404622691 0 0) 0x7fa408004eb0 con 0x7fa4240a9750 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4241dc1f0 con 0x7fa4240a9750 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa4200015b0 con 0x7fa4240a9750 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 10 monclient(hunting): dump: 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:04.701 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.705+0000 7fa41a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 10 client.?.objecter ms_handle_connect 0x7fa4240a9b30 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134723727 0 0) 0x7fa424071960 con 0x7fa4240a9b30 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.706+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa408004eb0 con 0x7fa4240a9b30 2024-03-29T11:30:04.702 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4206745462 0 0) 0x7fa408004eb0 con 0x7fa4240a9b30 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa424071960 con 0x7fa4240a9b30 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa4140017d0 con 0x7fa4240a9b30 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 10 monclient(hunting): dump: 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:04.703 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:04.704 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 10 client.?.objecter ms_handle_connect 0x7fa4240aad20 2024-03-29T11:30:04.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.707+0000 7fa41a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:04.704 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.708+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108893545 0 0) 0x7fa4241dc1f0 con 0x7fa4240a9750 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 >> v1:172.21.15.62:6789/0 conn(0x7fa4240a9b30 legacy=0x7fa4241deb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 >> v1:172.21.15.19:6789/0 conn(0x7fa4240aad20 legacy=0x7fa4241df280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 1 monclient: found mon.c 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4241d0860 con 0x7fa4240a9750 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:30:04.705 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 10 monclient: _renew_subs 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4241d0ab0 con 0x7fa4240a9750 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 10 monclient: _renew_subs 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4241e70c0 con 0x7fa4240a9750 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa42a87b640 1 librados: init done 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4200038a0 con 0x7fa4240a9750 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fa420003c30 con 0x7fa4240a9750 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:04.706 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.709+0000 7fa41a7fc640 10 monclient: dump: 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.710+0000 7fa41bfff640 4 set_mon_vals no callback set 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.711+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fa420003d90 con 0x7fa4240a9750 2024-03-29T11:30:04.707 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.711+0000 7fa42a87b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.711+0000 7fa42a87b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.711+0000 7fa42a87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa40c005080 con 0x7fa4240a9750 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa429078640 10 monclient: get_auth_request con 0x7fa40806de80 auth_method 0 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa41a7fc640 10 client.4661.objecter ms_handle_connect 0x7fa40806de80 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7fa420084c40 con 0x7fa4240a9750 2024-03-29T11:30:04.708 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa41a7fc640 10 client.4661.objecter ms_dispatch 0x7fa424000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:04.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa41a7fc640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:04.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa41a7fc640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:04.709 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.712+0000 7fa41a7fc640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:30:04.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.720+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa420050520 con 0x7fa4240a9750 2024-03-29T11:30:04.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.720+0000 7fa41a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:04.712 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:04.720+0000 7fa41a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:05.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.044+0000 7fa42a87b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:05.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.044+0000 7fa42a87b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:05.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.044+0000 7fa42a87b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.19:6790/0 2024-03-29T11:30:05.037 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.044+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 --> v1:172.21.15.19:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa40c005330 con 0x7fa4240a9750 2024-03-29T11:30:05.039 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.047+0000 7fa41a7fc640 1 -- 172.21.15.19:0/3529016279 <== mon.2 v1:172.21.15.19:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7fa420052450 con 0x7fa4240a9750 2024-03-29T11:30:05.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.047+0000 7fa41a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:05.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.047+0000 7fa41a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:05.043 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:30:05.043 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:05.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 10 librados: watch_flush enter 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 10 librados: watch_flush exit 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 >> v1:172.21.15.19:6816/3721232097 conn(0x7fa40806de80 legacy=0x7fa408070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 10 monclient: shutdown 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:05.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 >> v1:172.21.15.19:6790/0 conn(0x7fa4240a9750 legacy=0x7fa4241db8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:05.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.052+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 shutdown_connections 2024-03-29T11:30:05.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.053+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 >> 172.21.15.19:0/3529016279 conn(0x7fa4240673e0 msgr2=0x7fa4240ac340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:05.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.053+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 shutdown_connections 2024-03-29T11:30:05.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.053+0000 7fa42a87b640 1 -- 172.21.15.19:0/3529016279 wait complete. 2024-03-29T11:30:05.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:05.053+0000 7fa42a87b640 1 librados: shutdown 2024-03-29T11:30:05.066 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:11.874 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.026+0000 7fc3df59e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.026+0000 7fc3df59e640 10 monclient: build_initial_monmap 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.026+0000 7fc3df59e640 10 monclient: monmap: 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:30:12.028079+0000 2024-03-29T11:30:12.019 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:30:12.028079+0000 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.020 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.027+0000 7fc3df59e640 1 Processor -- start 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.027+0000 7fc3df59e640 1 -- start start 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.027+0000 7fc3df59e640 10 monclient: init 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e006ade0 addr v1:172.21.15.19:6789/0 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e0072090 addr v1:172.21.15.62:6789/0 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e014a110 addr v1:172.21.15.19:6790/0 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e006fdb0 con 0x7fc3e006ade0 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e0144010 con 0x7fc3e0072090 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e006eb00 con 0x7fc3e014a110 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:12.023 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.028+0000 7fc3df59e640 10 monclient(hunting): authenticate will time out at 1532.292663s 2024-03-29T11:30:12.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3ddd9b640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc3e0072090 0x7fc3e0149a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:51710/0 (socket says 172.21.15.19:51710) 2024-03-29T11:30:12.025 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3ddd9b640 1 -- 172.21.15.19:0/717877716 learned_addr learned my addr 172.21.15.19:0/717877716 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377387008 0 0) 0x7fc3e006eb00 con 0x7fc3e014a110 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3cc003520 con 0x7fc3e014a110 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386591840 0 0) 0x7fc3e0144010 con 0x7fc3e0072090 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e006eb00 con 0x7fc3e0072090 2024-03-29T11:30:12.026 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.032+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1147097705 0 0) 0x7fc3cc003520 con 0x7fc3e014a110 2024-03-29T11:30:12.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.033+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 >> v1:172.21.15.62:6789/0 conn(0x7fc3e0072090 legacy=0x7fc3e0149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:12.027 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.033+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 >> v1:172.21.15.19:6789/0 conn(0x7fc3e006ade0 legacy=0x7fc3e0071980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:12.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.035+0000 7fc3dd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:12.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.035+0000 7fc3dd59a640 1 monclient: found mon.noname-b 2024-03-29T11:30:12.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.035+0000 7fc3dd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:12.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.035+0000 7fc3dd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.19:6790/0 2024-03-29T11:30:12.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.035+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 --> v1:172.21.15.19:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e0070090 con 0x7fc3e014a110 2024-03-29T11:30:12.029 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 10 monclient: _finish_auth 0 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3d0001990 con 0x7fc3e014a110 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3dd59a640 10 monclient: dump: 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.030 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3df59e640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:30:12.031 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.036+0000 7fc3df59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:12.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 <== mon.2 v1:172.21.15.19:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3d0001bb0 con 0x7fc3e014a110 2024-03-29T11:30:12.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3dd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:12.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3df59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:12.033 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3df59e640 4 set_mon_vals no callback set 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3dd59a640 1 -- 172.21.15.19:0/717877716 <== mon.2 v1:172.21.15.19:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3d0001f40 con 0x7fc3e014a110 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3dd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3dd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.037+0000 7fc3dd59a640 10 monclient: dump: 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:12.034 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.038+0000 7fc3df59e640 10 monclient: shutdown 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.038+0000 7fc3df59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.038+0000 7fc3df59e640 1 -- 172.21.15.19:0/717877716 >> v1:172.21.15.19:6790/0 conn(0x7fc3e014a110 legacy=0x7fc3e0151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.038+0000 7fc3df59e640 1 -- 172.21.15.19:0/717877716 shutdown_connections 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.038+0000 7fc3df59e640 1 -- 172.21.15.19:0/717877716 >> 172.21.15.19:0/717877716 conn(0x7fc3e0067420 msgr2=0x7fc3e00b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.039+0000 7fc3df59e640 1 -- 172.21.15.19:0/717877716 shutdown_connections 2024-03-29T11:30:12.035 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.039+0000 7fc3df59e640 1 -- 172.21.15.19:0/717877716 wait complete. 2024-03-29T11:30:12.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 10 monclient: build_initial_monmap 2024-03-29T11:30:12.040 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 10 monclient: monmap: 2024-03-29T11:30:12.040 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:30:12.040 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:12.040 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:30:12.040 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 1 librados: starting msgr at 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 1 librados: starting objecter 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 1 Processor -- start 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 1 -- start start 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 1 librados: setting wanted keys 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 1 librados: calling monclient init 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.041+0000 7fc3df59e640 10 monclient: init 2024-03-29T11:30:12.041 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e006ade0 addr v1:172.21.15.19:6790/0 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e0072090 addr v1:172.21.15.19:6789/0 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e014a110 addr v1:172.21.15.62:6789/0 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 1 -- --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e00aa150 con 0x7fc3e0072090 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 1 -- --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e00ad650 con 0x7fc3e014a110 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 1 -- --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e00ad7e0 con 0x7fc3e006ade0 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3df59e640 10 monclient(hunting): authenticate will time out at 1532.306598s 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3ded9d640 1 --1- >> v1:172.21.15.62:6789/0 conn(0x7fc3e014a110 0x7fc3e01dbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.62:6789/0 says I am v1:172.21.15.19:52770/0 (socket says 172.21.15.19:52770) 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.042+0000 7fc3ded9d640 1 -- 172.21.15.19:0/727484443 learned_addr learned my addr 172.21.15.19:0/727484443 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.043+0000 7fc3d6ffd640 10 client.?.objecter ms_handle_connect 0x7fc3e014a110 2024-03-29T11:30:12.042 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.043+0000 7fc3d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.044+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257120723 0 0) 0x7fc3e00ad650 con 0x7fc3e014a110 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.044+0000 7fc3d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.044+0000 7fc3d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.044+0000 7fc3d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.044+0000 7fc3d6ffd640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.044+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0004eb0 con 0x7fc3e014a110 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 710268274 0 0) 0x7fc3c0004eb0 con 0x7fc3e014a110 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e00ad650 con 0x7fc3e014a110 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3d0001ec0 con 0x7fc3e014a110 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:12.043 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.045+0000 7fc3d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 client.?.objecter ms_handle_connect 0x7fc3e006ade0 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 client.?.objecter ms_handle_connect 0x7fc3e0072090 2024-03-29T11:30:12.044 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.2 v1:172.21.15.19:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843803042 0 0) 0x7fc3e00ad7e0 con 0x7fc3e006ade0 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.19:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c0004eb0 con 0x7fc3e006ade0 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900363173 0 0) 0x7fc3e00aa150 con 0x7fc3e0072090 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3e00ad7e0 con 0x7fc3e0072090 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.2 v1:172.21.15.19:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3737728343 0 0) 0x7fc3c0004eb0 con 0x7fc3e006ade0 2024-03-29T11:30:12.045 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.19:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3e00aa150 con 0x7fc3e006ade0 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.2 v1:172.21.15.19:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3d8001790 con 0x7fc3e006ade0 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:12.046 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.047 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3859331644 0 0) 0x7fc3e00ad7e0 con 0x7fc3e0072090 2024-03-29T11:30:12.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c0004eb0 con 0x7fc3e0072090 2024-03-29T11:30:12.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 649347695 0 0) 0x7fc3e00ad650 con 0x7fc3e014a110 2024-03-29T11:30:12.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 >> v1:172.21.15.19:6790/0 conn(0x7fc3e006ade0 legacy=0x7fc3e00acf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:12.048 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.046+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 >> v1:172.21.15.19:6789/0 conn(0x7fc3e0072090 legacy=0x7fc3e013f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 1 monclient: found mon.b 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e00adda0 con 0x7fc3e014a110 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.049+0000 7fc3d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3d0003c20 con 0x7fc3e014a110 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7fc3d0003f70 con 0x7fc3e014a110 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:12.049 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3d6ffd640 10 monclient: dump: 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.050+0000 7fc3dcd99640 4 set_mon_vals no callback set 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 5 monclient: authenticate success, global_id 4576 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 10 monclient: _renew_subs 2024-03-29T11:30:12.050 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e00adff0 con 0x7fc3e014a110 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 10 monclient: _renew_subs 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e013fb00 con 0x7fc3e014a110 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.051+0000 7fc3df59e640 1 librados: init done 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.053+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7fc3d00040d0 con 0x7fc3e014a110 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7fc3d0084d90 con 0x7fc3e014a110 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3d6ffd640 10 client.4576.objecter ms_dispatch 0x7fc3e0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3d6ffd640 3 client.4576.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3d6ffd640 3 client.4576.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3d6ffd640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3df59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:12.051 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3df59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:12.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3df59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:30:12.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.054+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e004f730 con 0x7fc3e014a110 2024-03-29T11:30:12.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.055+0000 7fc3de59c640 10 monclient: get_auth_request con 0x7fc3c006e040 auth_method 0 2024-03-29T11:30:12.052 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.055+0000 7fc3d6ffd640 10 client.4576.objecter ms_handle_connect 0x7fc3c006e040 2024-03-29T11:30:12.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.063+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3d0050670 con 0x7fc3e014a110 2024-03-29T11:30:12.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.063+0000 7fc3d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:12.056 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.063+0000 7fc3d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:12.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.399+0000 7fc3df59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:12.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.399+0000 7fc3df59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:12.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.399+0000 7fc3df59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.62:6789/0 2024-03-29T11:30:12.393 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.399+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 --> v1:172.21.15.62:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3e00b2f00 con 0x7fc3e014a110 2024-03-29T11:30:12.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.402+0000 7fc3d6ffd640 1 -- 172.21.15.19:0/727484443 <== mon.1 v1:172.21.15.62:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7fc3d00525a0 con 0x7fc3e014a110 2024-03-29T11:30:12.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.403+0000 7fc3d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:12.396 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.403+0000 7fc3d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:12.396 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:30:12.397 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:12.399 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.406+0000 7fc3df59e640 10 librados: watch_flush enter 2024-03-29T11:30:12.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 10 librados: watch_flush exit 2024-03-29T11:30:12.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-29T11:30:12.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 10 client.4576.objecter successfully canceled tick 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 >> v1:172.21.15.19:6816/3721232097 conn(0x7fc3c006e040 legacy=0x7fc3c0070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 10 monclient: shutdown 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 >> v1:172.21.15.62:6789/0 conn(0x7fc3e014a110 legacy=0x7fc3e01dbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 shutdown_connections 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 >> 172.21.15.19:0/727484443 conn(0x7fc3e0067420 msgr2=0x7fc3e00af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.409+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 shutdown_connections 2024-03-29T11:30:12.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.410+0000 7fc3df59e640 1 -- 172.21.15.19:0/727484443 wait complete. 2024-03-29T11:30:12.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:12.410+0000 7fc3df59e640 1 librados: shutdown 2024-03-29T11:30:12.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:19.231 DEBUG:teuthology.orchestra.run.smithi019:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:19.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.386+0000 7f6a5d9ba640 10 monclient: get_monmap_and_config 2024-03-29T11:30:19.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.386+0000 7f6a5d9ba640 10 monclient: build_initial_monmap 2024-03-29T11:30:19.380 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.386+0000 7f6a5d9ba640 10 monclient: monmap: 2024-03-29T11:30:19.380 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:30:19.388279+0000 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:30:19.388279+0000 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:30:19.381 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-c 2024-03-29T11:30:19.382 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-b 2024-03-29T11:30:19.382 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:19.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.387+0000 7f6a5d9ba640 1 Processor -- start 2024-03-29T11:30:19.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.387+0000 7f6a5d9ba640 1 -- start start 2024-03-29T11:30:19.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.387+0000 7f6a5d9ba640 10 monclient: init 2024-03-29T11:30:19.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.388+0000 7f6a5d9ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:19.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.388+0000 7f6a5d9ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:19.385 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.388+0000 7f6a5d9ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6a581457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:30:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.388+0000 7f6a5d9ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6a5813f650 addr v1:172.21.15.19:6789/0 2024-03-29T11:30:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.388+0000 7f6a4ffff640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f6a5813f650 0x7f6a580b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:46232/0 (socket says 172.21.15.19:46232) 2024-03-29T11:30:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.388+0000 7f6a4ffff640 1 -- 172.21.15.19:0/3012808995 learned_addr learned my addr 172.21.15.19:0/3012808995 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:30:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.389+0000 7f6a5d9ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6a580a9d70 addr v1:172.21.15.19:6790/0 2024-03-29T11:30:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.389+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a580aaef0 con 0x7f6a5813f650 2024-03-29T11:30:19.386 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.389+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a58145bd0 con 0x7f6a581457f0 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.389+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a58145eb0 con 0x7f6a580a9d70 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.389+0000 7f6a5d9ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.389+0000 7f6a5d9ba640 10 monclient(hunting): authenticate will time out at 1539.653804s 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.390+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952694899 0 0) 0x7f6a580aaef0 con 0x7f6a5813f650 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.390+0000 7f6a567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.390+0000 7f6a567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:19.387 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.390+0000 7f6a567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:19.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.390+0000 7f6a567fc640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:30:19.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.390+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a44003520 con 0x7f6a5813f650 2024-03-29T11:30:19.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.391+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3891682528 0 0) 0x7f6a44003520 con 0x7f6a5813f650 2024-03-29T11:30:19.388 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.391+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 >> v1:172.21.15.19:6790/0 conn(0x7f6a580a9d70 legacy=0x7f6a5814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:19.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 >> v1:172.21.15.62:6789/0 conn(0x7f6a581457f0 legacy=0x7f6a5813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:19.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:19.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 1 monclient: found mon.noname-a 2024-03-29T11:30:19.402 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:19.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.19:6789/0 2024-03-29T11:30:19.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58146340 con 0x7f6a5813f650 2024-03-29T11:30:19.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:19.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:19.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:19.403 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6a40001690 con 0x7f6a5813f650 2024-03-29T11:30:19.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:19.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:19.404 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.392+0000 7f6a567fc640 10 monclient: dump: 2024-03-29T11:30:19.404 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:19.404 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:19.404 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.405 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.405 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:19.405 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:19.405 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:19.405 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:19.405 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:19.406 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:19.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a5d9ba640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:30:19.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a5d9ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:19.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 <== mon.0 v1:172.21.15.19:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a40001860 con 0x7f6a5813f650 2024-03-29T11:30:19.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:19.406 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a567fc640 1 -- 172.21.15.19:0/3012808995 <== mon.0 v1:172.21.15.19:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6a40001cb0 con 0x7f6a5813f650 2024-03-29T11:30:19.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:19.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:19.407 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a567fc640 10 monclient: dump: 2024-03-29T11:30:19.407 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:19.407 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:19.407 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.408 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.408 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:19.408 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:19.408 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:19.408 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:19.408 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:19.409 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:19.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a5d9ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:19.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.393+0000 7f6a5d9ba640 4 set_mon_vals no callback set 2024-03-29T11:30:19.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 10 monclient: shutdown 2024-03-29T11:30:19.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:19.409 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 >> v1:172.21.15.19:6789/0 conn(0x7f6a5813f650 legacy=0x7f6a580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:19.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 shutdown_connections 2024-03-29T11:30:19.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 >> 172.21.15.19:0/3012808995 conn(0x7f6a58066a00 msgr2=0x7f6a58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:19.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 shutdown_connections 2024-03-29T11:30:19.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.394+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/3012808995 wait complete. 2024-03-29T11:30:19.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.395+0000 7f6a5d9ba640 10 monclient: build_initial_monmap 2024-03-29T11:30:19.410 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.396+0000 7f6a5d9ba640 10 monclient: monmap: 2024-03-29T11:30:19.411 INFO:teuthology.misc.health.smithi019.stderr:epoch 0 2024-03-29T11:30:19.411 INFO:teuthology.misc.health.smithi019.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:19.411 INFO:teuthology.misc.health.smithi019.stderr:last_changed 0.000000 2024-03-29T11:30:19.411 INFO:teuthology.misc.health.smithi019.stderr:created 0.000000 2024-03-29T11:30:19.411 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:19.412 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:19.412 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.noname-a 2024-03-29T11:30:19.412 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.noname-b 2024-03-29T11:30:19.412 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.noname-c 2024-03-29T11:30:19.412 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:19.412 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.396+0000 7f6a5d9ba640 1 librados: starting msgr at 2024-03-29T11:30:19.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.396+0000 7f6a5d9ba640 1 librados: starting objecter 2024-03-29T11:30:19.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.396+0000 7f6a5d9ba640 1 Processor -- start 2024-03-29T11:30:19.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.397+0000 7f6a5d9ba640 1 -- start start 2024-03-29T11:30:19.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.397+0000 7f6a5d9ba640 1 librados: setting wanted keys 2024-03-29T11:30:19.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.397+0000 7f6a5d9ba640 1 librados: calling monclient init 2024-03-29T11:30:19.413 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.397+0000 7f6a5d9ba640 10 monclient: init 2024-03-29T11:30:19.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.397+0000 7f6a5d9ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:19.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:19.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6a580a9d70 addr v1:172.21.15.19:6789/0 2024-03-29T11:30:19.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a56ffd640 1 --1- >> v1:172.21.15.19:6789/0 conn(0x7f6a580a9d70 0x7f6a58140330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.19:6789/0 says I am v1:172.21.15.19:59726/0 (socket says 172.21.15.19:59726) 2024-03-29T11:30:19.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a56ffd640 1 -- 172.21.15.19:0/192912747 learned_addr learned my addr 172.21.15.19:0/192912747 (peer_addr_for_me v1:172.21.15.19:0/0) 2024-03-29T11:30:19.414 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6a5813f650 addr v1:172.21.15.19:6790/0 2024-03-29T11:30:19.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 10 monclient(hunting): picked mon.noname-b con 0x7f6a581457f0 addr v1:172.21.15.62:6789/0 2024-03-29T11:30:19.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a58141910 con 0x7f6a580a9d70 2024-03-29T11:30:19.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.62:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5813b1b0 con 0x7f6a581457f0 2024-03-29T11:30:19.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a58142cb0 con 0x7f6a5813f650 2024-03-29T11:30:19.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:19.415 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a5d9ba640 10 monclient(hunting): authenticate will time out at 1539.662476s 2024-03-29T11:30:19.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 10 client.?.objecter ms_handle_connect 0x7f6a580a9d70 2024-03-29T11:30:19.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:19.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174511954 0 0) 0x7f6a58141910 con 0x7f6a580a9d70 2024-03-29T11:30:19.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:30:19.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:19.416 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:19.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:30:19.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.398+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a38004eb0 con 0x7f6a580a9d70 2024-03-29T11:30:19.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1723017700 0 0) 0x7f6a38004eb0 con 0x7f6a580a9d70 2024-03-29T11:30:19.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a58141910 con 0x7f6a580a9d70 2024-03-29T11:30:19.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6a480018f0 con 0x7f6a580a9d70 2024-03-29T11:30:19.417 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 10 monclient(hunting): dump: 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.418 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1876193517 0 0) 0x7f6a58141910 con 0x7f6a580a9d70 2024-03-29T11:30:19.419 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.399+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 >> v1:172.21.15.19:6790/0 conn(0x7f6a5813f650 legacy=0x7f6a581425a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 >> v1:172.21.15.62:6789/0 conn(0x7f6a581457f0 legacy=0x7f6a581e1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 1 monclient: found mon.a 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5813fda0 con 0x7f6a580a9d70 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: _finish_auth 0 2024-03-29T11:30:19.420 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 10 monclient: _renew_subs 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a581e3050 con 0x7f6a580a9d70 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 client.4596.objecter ms_handle_connect 0x7f6a581457f0 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 client.4596.objecter resend_mon_ops 2024-03-29T11:30:19.421 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:19.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 10 monclient: _renew_subs 2024-03-29T11:30:19.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:30:19.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a581e33c0 con 0x7f6a580a9d70 2024-03-29T11:30:19.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a5d9ba640 1 librados: init done 2024-03-29T11:30:19.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a48001d40 con 0x7f6a580a9d70 2024-03-29T11:30:19.422 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 897219370 0 0) 0x7f6a48003a60 con 0x7f6a580a9d70 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a54ff9640 10 monclient: dump: 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:epoch 1 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:fsid 7173a58a-f9c8-420e-9665-277a04c8f803 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:last_changed 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:created 2024-03-29T11:18:43.591240+0000 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:19.423 INFO:teuthology.misc.health.smithi019.stderr:election_strategy: 1 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:0: v1:172.21.15.19:6789/0 mon.a 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:1: v1:172.21.15.62:6789/0 mon.b 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2: v1:172.21.15.19:6790/0 mon.c 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr: 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.401+0000 7f6a55ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.403+0000 7f6a5d9ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.403+0000 7f6a5d9ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.403+0000 7f6a5d9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.403+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3c005080 con 0x7f6a580a9d70 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.403+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3126214997 0 0) 0x7f6a48003bc0 con 0x7f6a580a9d70 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.404+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1639661479 0 0) 0x7f6a48084a50 con 0x7f6a580a9d70 2024-03-29T11:30:19.424 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.404+0000 7f6a54ff9640 10 client.4596.objecter ms_dispatch 0x7f6a58000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:19.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.404+0000 7f6a54ff9640 3 client.4596.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:19.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.404+0000 7f6a54ff9640 3 client.4596.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:19.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.404+0000 7f6a54ff9640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:30:19.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.405+0000 7f6a4ffff640 10 monclient: get_auth_request con 0x7f6a3806db80 auth_method 0 2024-03-29T11:30:19.425 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.405+0000 7f6a54ff9640 10 client.4596.objecter ms_handle_connect 0x7f6a3806db80 2024-03-29T11:30:19.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.412+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a48050330 con 0x7f6a580a9d70 2024-03-29T11:30:19.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.412+0000 7f6a54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:19.426 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.412+0000 7f6a54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:19.729 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.736+0000 7f6a5d9ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:19.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.736+0000 7f6a5d9ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:19.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.736+0000 7f6a5d9ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.19:6789/0 2024-03-29T11:30:19.730 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.736+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 --> v1:172.21.15.19:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a3c005330 con 0x7f6a580a9d70 2024-03-29T11:30:19.730 INFO:teuthology.misc.health.smithi019.stdout: 2024-03-29T11:30:19.731 INFO:teuthology.misc.health.smithi019.stdout:{"epoch":11,"fsid":"7173a58a-f9c8-420e-9665-277a04c8f803","created":"2024-03-29T11:19:21.804408+0000","modified":"2024-03-29T11:29:26.752886+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:29:26.752886+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e330fe46-e3ef-468b-8a2e-b8c79f658692","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"73c12c41-ab79-4175-80b3-80ebd03e7499","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"fc3f3fea-ba5e-4e6a-9313-446861b7a8a1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"1cdeb73f-e76f-495b-bb9e-8ef1f72d96c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"70842ebb-d0e8-4235-9ac9-dad2b393a0f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"57b4c2d1-f924-44f9-843c-a9eb27b02126","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"f5fbd5e2-d81e-4de1-8135-d2e9a6ae5734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ecb106b2-c5e5-4ff8-ac8e-49090ad27960","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:19.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.737+0000 7f6a54ff9640 1 -- 172.21.15.19:0/192912747 <== mon.0 v1:172.21.15.19:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3816626840) 0x7f6a48052260 con 0x7f6a580a9d70 2024-03-29T11:30:19.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.737+0000 7f6a54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:19.731 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.737+0000 7f6a54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:19.733 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.740+0000 7f6a5d9ba640 10 librados: watch_flush enter 2024-03-29T11:30:19.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.740+0000 7f6a5d9ba640 10 librados: watch_flush exit 2024-03-29T11:30:19.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.740+0000 7f6a5d9ba640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:30:19.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.740+0000 7f6a5d9ba640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:30:19.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 >> v1:172.21.15.19:6816/3721232097 conn(0x7f6a3806db80 legacy=0x7f6a38070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:19.734 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 10 monclient: shutdown 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 >> v1:172.21.15.19:6789/0 conn(0x7f6a580a9d70 legacy=0x7f6a58140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 shutdown_connections 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 >> 172.21.15.19:0/192912747 conn(0x7f6a58066a00 msgr2=0x7f6a580b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 shutdown_connections 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 -- 172.21.15.19:0/192912747 wait complete. 2024-03-29T11:30:19.735 INFO:teuthology.misc.health.smithi019.stderr:2024-03-29T11:30:19.741+0000 7f6a5d9ba640 1 librados: shutdown 2024-03-29T11:30:19.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:19.752 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:30:19.754 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:30:19.754 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:30:19.754 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:19.804 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:30:19.804 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:30:19.804 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:19.842 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:30:19.843 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:30:19.843 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:19.914 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:30:19.914 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:30:19.914 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:19.982 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:30:19.982 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:30:19.982 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.021 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:30:20.021 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:30:20.021 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.089 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:30:20.089 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:30:20.089 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.164 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:30:20.164 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:30:20.164 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.233 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:30:20.233 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:30:20.233 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:30:20.234 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.258 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:30:20.258 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:30:20.259 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.298 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:30:20.301 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:30:20.302 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:30:20.302 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.313 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:30:20.313 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:30:20.313 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.365 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:30:20.365 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:30:20.365 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:30:20.375 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:30:20.376 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:30:20.376 DEBUG:teuthology.orchestra.run.smithi019:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:30:20.482 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:30:20.482 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:30:24.458 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:30:24.458 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:30:24.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:30:24.571 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:30:24.661 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:30:24.662 DEBUG:teuthology.orchestra.run.smithi019:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:30:24.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:30:24.729 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:30:26.619 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:30:26.619 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:30:26.732 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:30:26.732 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:30:26.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:30:26.779 DEBUG:teuthology.orchestra.run.smithi062:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:30:26.831 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:30:26.832 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/data/mon.a.tgz 2024-03-29T11:30:26.833 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-29T11:30:26.851 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.yIDyPhRp2B 2024-03-29T11:30:26.852 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.yIDyPhRp2B 2024-03-29T11:30:26.951 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.yIDyPhRp2B 2024-03-29T11:30:27.039 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.yIDyPhRp2B is 126KB 2024-03-29T11:30:27.091 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.yIDyPhRp2B 2024-03-29T11:30:27.106 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/data/mon.c.tgz 2024-03-29T11:30:27.106 DEBUG:teuthology.orchestra.run.smithi019:> mktemp 2024-03-29T11:30:27.161 INFO:teuthology.orchestra.run.smithi019.stdout:/tmp/tmp.GDaCdfPuC3 2024-03-29T11:30:27.162 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.GDaCdfPuC3 2024-03-29T11:30:27.250 DEBUG:teuthology.orchestra.run.smithi019:> sudo chmod 0666 /tmp/tmp.GDaCdfPuC3 2024-03-29T11:30:27.336 DEBUG:teuthology.orchestra.remote:smithi019:/tmp/tmp.GDaCdfPuC3 is 126KB 2024-03-29T11:30:27.387 DEBUG:teuthology.orchestra.run.smithi019:> rm -fr /tmp/tmp.GDaCdfPuC3 2024-03-29T11:30:27.401 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/data/mon.b.tgz 2024-03-29T11:30:27.401 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-03-29T11:30:27.419 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.2Lu2oIZldg 2024-03-29T11:30:27.419 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.2Lu2oIZldg 2024-03-29T11:30:27.521 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.2Lu2oIZldg 2024-03-29T11:30:27.609 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.2Lu2oIZldg is 126KB 2024-03-29T11:30:27.661 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.2Lu2oIZldg 2024-03-29T11:30:27.676 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:30:27.677 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:27.682 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:30:27.760 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:27.800 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:30:27.800 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:30:27.802 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/remote/smithi019/crash 2024-03-29T11:30:27.803 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:27.830 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/remote/smithi062/crash 2024-03-29T11:30:27.831 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:30:27.868 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:30:27.868 DEBUG:teuthology.orchestra.run.smithi019:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:27.875 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39882.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39882.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28723.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39063.log 2024-03-29T11:30:27.932 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39882.log.gz 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.tmp-client.admin.28723.log: 0.0%gzip -- replaced with /var/log/ceph/ceph.tmp-client.admin.28723.log.gz -5 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41149.log 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39531.log 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39063.log.gz 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40369.log 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41149.log.gz 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40369.log: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.39531.log.gz 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41500.log 2024-03-29T11:30:27.933 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40876.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38946.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38790.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39999.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41500.log.gz 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39765.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38946.log.gz 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39843.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38790.log.gz 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39999.log.gz 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5/var/log/ceph/ceph-client.admin.39765.log: --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-03-29T11:30:27.934 INFO:teuthology.orchestra.run.smithi019.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39765.log.gz 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 85.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39843.log.gz 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/ceph-client.admin.39687.log 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41617.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39219.log 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39453.log 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38516.log 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39687.log.gz 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39219.log.gz 2024-03-29T11:30:27.935 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.39453.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41188.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41227.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.39453.log.gz 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39024.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38516.log.gz 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41227.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41188.log.gz 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41227.log.gz 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41539.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39024.log.gz 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40447.log 2024-03-29T11:30:27.936 INFO:teuthology.orchestra.run.smithi019.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38868.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gz 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34542.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38868.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39609.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38868.log.gz 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34542.log: gzip 87.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz --verbose 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr: -- /var/log/ceph/ceph-client.admin.40603.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.34542.log.gz 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34936.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38712.log 2024-03-29T11:30:27.937 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40603.log: /var/log/ceph/ceph-client.admin.34936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39609.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39648.log 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34936.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38241.log 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38712.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41578.log 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38241.log: 86.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.39648.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41578.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38241.log.gz 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38634.log 2024-03-29T11:30:27.938 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41578.log.gz 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38634.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40837.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38634.log.gz 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40173.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38556.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39921.log: /var/log/ceph/ceph-client.admin.38556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39180.log 2024-03-29T11:30:27.939 INFO:teuthology.orchestra.run.smithi019.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41695.log 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38556.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39180.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41695.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41266.log 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38475.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40681.log 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41266.log: gzip 86.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz -- 2024-03-29T11:30:27.940 INFO:teuthology.orchestra.run.smithi019.stderr: /var/log/ceph/ceph-client.admin.40759.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40681.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.41266.log.gz 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40759.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41383.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.3.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38985.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41383.log: /var/log/ceph/ceph-client.admin.38985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38280.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41383.log.gz 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41344.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38985.log.gz 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38280.log: gzip -5 --verbose -- 85.7% /var/log/ceph/ceph-client.admin.38751.log 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-29T11:30:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39570.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38280.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40134.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41344.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40330.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38751.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39570.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.40330.log: -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40134.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39336.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39141.log 2024-03-29T11:30:27.942 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38319.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39804.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39336.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39297.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39141.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39258.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39804.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38319.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39297.log: gzip -5 --verbose 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39804.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39297.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38358.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40213.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39258.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38358.log: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-29T11:30:27.943 INFO:teuthology.orchestra.run.smithi019.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.38358.log.gz 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39414.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38595.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39414.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38595.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39414.log.gz 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38907.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38595.log.gz 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41305.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.41305.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38907.log.gz 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41305.log.gz 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:30:27.944 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40915.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-03-29T11:30:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40564.log 2024-03-29T11:30:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34833.log 2024-03-29T11:30:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:gzip/var/log/ceph/ceph-client.admin.40564.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39726.log 2024-03-29T11:30:27.945 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.34833.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-03-29T11:30:27.945 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34833.log.gz 2024-03-29T11:30:27.958 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:30:27.958 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34781.log 2024-03-29T11:30:27.958 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34821.log 2024-03-29T11:30:27.958 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34741.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34821.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34781.log.gz 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34621.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34821.log.gz 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.audit.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34741.log.gz 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34621.log: 88.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34701.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34621.log.gz 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34701.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34701.log.gz 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:30:27.959 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34661.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34581.log 87.7% 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-client.admin.34581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34541.log 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34581.log.gz 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34541.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34541.log.gz 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi062.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:30:27.960 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39492.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39492.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39726.log.gz 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39492.log.gz 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38829.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39102.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38829.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39375.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38829.log.gz 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38436.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.39375.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39102.log.gz 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:gzip -5 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39375.log.gz 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40095.log 2024-03-29T11:30:27.961 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40054.log 2024-03-29T11:30:27.962 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40095.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38436.log.gz 2024-03-29T11:30:27.962 INFO:teuthology.orchestra.run.smithi019.stderr: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:30:27.962 INFO:teuthology.orchestra.run.smithi019.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40095.log.gz 2024-03-29T11:30:27.962 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.40054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2024-03-29T11:30:27.962 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-mon.c.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40054.log.gz 2024-03-29T11:30:27.962 INFO:teuthology.orchestra.run.smithi019.stderr:/var/log/ceph/ceph-client.admin.38397.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2024-03-29T11:30:27.990 INFO:teuthology.orchestra.run.smithi019.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:30:39.629 INFO:teuthology.orchestra.run.smithi062.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:30:39.629 INFO:teuthology.orchestra.run.smithi062.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:30:39.667 INFO:teuthology.orchestra.run.smithi062.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:30:39.786 INFO:teuthology.orchestra.run.smithi019.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:30:43.088 INFO:teuthology.orchestra.run.smithi019.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:30:43.766 INFO:teuthology.orchestra.run.smithi062.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:30:44.445 INFO:teuthology.orchestra.run.smithi019.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:30:44.824 INFO:teuthology.orchestra.run.smithi019.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:31:15.338 INFO:teuthology.orchestra.run.smithi019.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:31:22.720 INFO:teuthology.orchestra.run.smithi019.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:31:22.722 INFO:teuthology.orchestra.run.smithi019.stderr: 2024-03-29T11:31:22.722 INFO:teuthology.orchestra.run.smithi019.stderr:real 0m54.834s 2024-03-29T11:31:22.722 INFO:teuthology.orchestra.run.smithi019.stderr:user 2m37.386s 2024-03-29T11:31:22.722 INFO:teuthology.orchestra.run.smithi019.stderr:sys 0m5.151s 2024-03-29T11:31:43.048 INFO:teuthology.orchestra.run.smithi062.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:31:43.049 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-29T11:31:43.049 INFO:teuthology.orchestra.run.smithi062.stderr:real 1m15.118s 2024-03-29T11:31:43.050 INFO:teuthology.orchestra.run.smithi062.stderr:user 2m1.825s 2024-03-29T11:31:43.050 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m3.992s 2024-03-29T11:31:43.050 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:31:43.050 DEBUG:teuthology.misc:Transferring archived files from smithi019:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/remote/smithi019/log 2024-03-29T11:31:43.133 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:06.264 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/remote/smithi062/log 2024-03-29T11:32:06.265 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:32:25.152 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:25.357 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e4af4c598014afa9143cc2b7c6c608d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:32:25.361 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:32:25.374 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:32:25.388 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:32:25.388 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:25.404 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:32:25.479 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:32:25.490 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:32:25.491 DEBUG:teuthology.orchestra.run.smithi019:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:25.494 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:32:25.507 INFO:teuthology.orchestra.run.smithi019.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:25.510 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:32:25.512 INFO:teuthology.orchestra.run.smithi019.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:25.512 INFO:teuthology.orchestra.run.smithi019.stdout:=============================================================================== 2024-03-29T11:32:25.513 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 44 +458us[ +458us] +/- 91ms 2024-03-29T11:32:25.513 INFO:teuthology.orchestra.run.smithi019.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -117us[ -125us] +/- 54ms 2024-03-29T11:32:25.513 INFO:teuthology.orchestra.run.smithi019.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -262us[ -269us] +/- 86ms 2024-03-29T11:32:25.513 INFO:teuthology.orchestra.run.smithi019.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:25.514 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:32:25.514 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-03-29T11:32:25.514 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +471us[ +471us] +/- 91ms 2024-03-29T11:32:25.514 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -112us[ -117us] +/- 54ms 2024-03-29T11:32:25.515 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 46 -263us[ -269us] +/- 86ms 2024-03-29T11:32:25.515 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:32:25.516 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:32:25.576 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:32:25.576 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:32:25.588 DEBUG:teuthology.orchestra.run.smithi019:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:25.659 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:32:25.733 DEBUG:teuthology.orchestra.run.smithi019:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:25.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:25.765 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:32:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:25.796 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:32:25.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:32:25.820 INFO:teuthology.task.internal:Duration was 1312.341561 seconds 2024-03-29T11:32:25.821 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:32:25.832 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:32:25.832 DEBUG:teuthology.orchestra.run.smithi019:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:25.835 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:32:25.877 INFO:teuthology.orchestra.run.smithi019.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:25.882 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:32:26.211 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:32:26.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi019.front.sepia.ceph.com 2024-03-29T11:32:26.212 DEBUG:teuthology.orchestra.run.smithi019:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:26.232 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-03-29T11:32:26.232 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:32:26.267 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:32:26.268 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:26.274 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:32:26.338 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:32:26.338 DEBUG:teuthology.orchestra.run.smithi019:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:26.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:32:26.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:32:26.489 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:32:26.489 DEBUG:teuthology.orchestra.run.smithi019:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:26.517 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:32:26.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:32:26.554 DEBUG:teuthology.orchestra.run.smithi019:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:26.560 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:26.583 INFO:teuthology.orchestra.run.smithi019.stdout:kernel.core_pattern = core 2024-03-29T11:32:26.610 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-03-29T11:32:26.637 DEBUG:teuthology.orchestra.run.smithi019:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:26.658 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:32:26.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:32:26.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:32:26.701 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:32:26.702 DEBUG:teuthology.misc:Transferring archived files from smithi019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/remote/smithi019 2024-03-29T11:32:26.702 DEBUG:teuthology.orchestra.run.smithi019:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:26.738 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685/remote/smithi062 2024-03-29T11:32:26.738 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:32:26.777 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:32:26.778 DEBUG:teuthology.orchestra.run.smithi019:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:26.780 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:32:26.823 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:32:26.836 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:32:26.837 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:32:26.847 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:32:26.847 DEBUG:teuthology.orchestra.run.smithi019:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:26.849 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:32:26.861 INFO:teuthology.orchestra.run.smithi019.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:26.877 INFO:teuthology.orchestra.run.smithi062.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:32 /home/ubuntu/cephtest 2024-03-29T11:32:26.878 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:32:26.888 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:32:26.914 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:32:26.943 DEBUG:teuthology.nuke:shortname: smithi019 2024-03-29T11:32:26.944 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:26.970 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi019.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:05.708424', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGmST/JS+fhMJ4n0lfob5aItaHmAVSIgXeMLzvsxBGyZM1xYrIRbGnPndg+PnnS6xedpMCjj7De3Od6LJY6fDJo='} 2024-03-29T11:32:26.974 DEBUG:teuthology.nuke:shortname: smithi062 2024-03-29T11:32:26.974 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:32:27.000 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622685', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:05.710108', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAoMD5AnRoAI8E0Aydsy5qWPX4rpSqlIGnpKho++nzwrit0iEO/vBeomdfg2KdU0flCdTWjSvFvqagARkIeHhmc='} 2024-03-29T11:32:27.030 INFO:teuthology.orchestra.console.smithi019:Power off 2024-03-29T11:32:27.031 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:27.056 DEBUG:teuthology.orchestra.console.smithi019:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:27.056 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:27.074 INFO:teuthology.orchestra.console.smithi062:Power off 2024-03-29T11:32:27.075 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:32:27.086 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-29T11:32:27.102 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-03-29T11:32:27.103 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:27.129 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-29T11:32:31.088 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:31.130 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:31.201 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is on 2024-03-29T11:32:31.243 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-29T11:32:35.202 DEBUG:teuthology.orchestra.console.smithi019:pexpect command: ipmitool -H smithi019.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:35.244 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:35.313 DEBUG:teuthology.orchestra.console.smithi019:check power output: Chassis Power is off 2024-03-29T11:32:35.355 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-29T11:32:35.414 INFO:teuthology.orchestra.console.smithi019:Power off completed 2024-03-29T11:32:39.360 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:32:39.470 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-03-29T11:32:39.571 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-03-29T11:32:39.675 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1312.3415608406067 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8e4af4c598014afa9143cc2b7c6c608d status: fail success: false 2024-03-29T11:32:39.675 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:32:39.755 INFO:teuthology.run:FAIL